Consensus score

A measure of agreement among reviewers on an item or set of items.

The calculation depends on the task: it may summarize matching decisions or variation across scored dimensions. State the denominator, tolerance and treatment of missing judgments. A consensus score is not the same as a quality score or a statistically calibrated probability of correctness.

In practice

All three reviewers can agree that an item fails, producing strong consensus alongside a poor quality verdict.