Human in the loop

A workflow in which people contribute judgments or decisions to an automated process.

Human involvement can focus on defining criteria, producing reference judgments, resolving uncertain cases and checking continued performance. It need not mean reviewing every item manually. The workflow should specify who decides, what evidence they see and how disagreements are resolved.

In practice

Experts establish reference judgments and review cases that automated judges cannot resolve with sufficient consensus.