Active learning

A method that selects useful examples for labeling instead of treating every unlabeled item as equally valuable.

Selection can focus on uncertainty, disagreement or gaps in coverage. People label the selected examples, and the model is updated. Sampling for learning and sampling to estimate quality have different goals: a deliberately difficult sample does not represent the error rate of the whole dataset.

In practice

A robotics team prioritizes unfamiliar grasp configurations for annotation, while keeping a separate random sample for its quality estimate.