Annotation agreement

A measure of how consistently different reviewers label the same examples under the same instructions.

Agreement can reveal unclear categories or inconsistent interpretation. It does not establish correctness: reviewers may share the same mistake. Report which agreement measure was used, the number of reviewers and the kinds of data reviewed. Compare judgments with a defensible reference where one exists.

In practice

Three reviewers can agree that a clip shows success while all missing a dropped object at the end.