Blind expert spot check

A review of selected production items against expert answers that are hidden from the automated judges.

Blind checks test whether judges continue to match the intended standard during work. They should be indistinguishable from ordinary items to the judges. Random checks and checks targeted at known risks answer different questions. Report the selection method before generalizing the results to the whole dataset.

In practice

A randomly selected clip is also reviewed by experts; the judge's result is compared with their reference without revealing that it was a check.