Expert reference set

Examples with documented expert judgments that provide a reference for evaluating other reviewers or systems.

The judgments must use the same rubric as the work being assessed. Record how experts were selected, how disagreements were handled and what the sample covers. Some examples may be shared for development, while separate answers are withheld for evaluation or blind checks.

In practice

Judges are compared on unfamiliar clips whose expert answers are hidden from them.