Judge alignment

How closely a judge's evaluations match expert reference judgments under a particular rubric.

Alignment is measured on specified examples using a defined scoring rule. It may involve categorical agreement or tolerance on a scored dimension. State the sample size, coverage and uncertainty. Strong alignment on one dataset is not proof of general alignment with human intent or reliable performance on every future item.

In practice

Two judges with the same average score may differ sharply on occluded scenes, so the difficult slice is reported separately.