Annotation consistency

The degree to which the same labeling rules are applied across examples, reviewers and time.

Consistency depends on explicit instructions and stable interpretations of borderline cases. It is possible to be consistently wrong, so consistency checks should accompany checks against reference judgments. When a rubric changes, record its version so apparent changes in labeling can be explained.

In practice

The start of a grasp should mean the same event in every clip, rather than contact in some clips and closure in others.