We ran a trial on real ML training data from an automotive history platform. Proof of Quality found the bad labels, the models got better when we fixed them, and far less of the work needed a human.

The Takeaways:

• It catches the real errors. We hand checked the 32 labels that drew the most doubt. Nine were genuinely wrong, and human consensus flagged all nine while leaving all 23 correct labels alone. Nothing correct got flagged.
• Fixing them moved the models. Correcting nine labels lifted one model from 70% to 82% on the hard cases. A second, stronger model went from 86% to 93%.
• About five reviewers per item, at any scale. Only around 43% of a single score reflects the label itself, so consensus does the real work. It sharpens as 1/√n, and rare errors took about five independent reviewers to catch reliably.
• An AI grader alone gets it wrong, but it clears the queue. Most of what the AI panel flagged on its own turned out to be fine. Run it as a first pass instead and it cut human review volume by about 74% without losing a single error.

The trial

Vehicle listings arrive in rough shape. They come with spelling errors, abbreviations, fragments, duplicate terms, and stray text, and each one still has to match the right entry in a vehicle catalogue, where some entries differ by a single detail.

We ran a trial with an automotive history platform on 250 of these labels. Every messy listing already carried a human supplied match to a catalogue entry, and we wanted to answer one question: could Proof of Quality spot the labels that deserved another look?

Ten human reviewers scored each label, once with a small stake riding on the outcome and once without. A panel of eight AI reviewers scored the same work, and a second AI setup with six reviewers and revised instructions ran alongside it. Everyone scored independently, 0 to 100, against the same rubric. PoQ combined the scores into consensus results, and we then hand checked the 32 labels that drew the most doubt against the underlying vehicle information.

Those 32 labels are the verified core of this report. They came from one product family and one person did the hand check, so we keep the claims focused on the labels we actually checked.

It catches the real errors

The hand check found nine wrong labels and 23 correct ones. Human consensus separated them cleanly at the pass or flag threshold. All nine errors got flagged, all 23 correct labels stayed in place, and nothing correct got flagged along the way.

In other words, when the group raised doubt about a label, there was a real problem underneath. When it accepted one, the hand check backed the decision.

There is also a broader way to score agreement across all 32 labels. The human group with a small stake matched the hand check on 30 of 32, or 94%. The group without a stake matched on 29 of 32, or 91%.

Fixing nine labels moved the models

Finding bad labels only matters if fixing them changes what a model learns, so we tested exactly that. We trained models on the original data and on versions with the nine errors removed or corrected.

The first model started at 70% on the hard cases. Removing the nine bad labels took it to 80%, and correcting them took it to 82%, which is 12 percentage points from nine labels. A second, more advanced model moved the same way, from 86% to 93%.

These errors sat exactly where the task was hardest, among listings where catalogue entries look nearly identical and one vehicle detail decides the match. Fixing them cleaned up the training signal right where the models needed it.

About five reviewers per item, at any scale

A single review mixes real information with personal interpretation and plain randomness. Across every vote in the trial, only about 43% of an individual score reflected the label itself. The rest was reviewer differences and noise.

Consensus recovers the signal. When reviewers work independently, the shared evidence stacks up across their scores while individual quirks cancel out. In our data, consensus sharpened by the familiar 1/√n law. Rare errors took about five reviewers to catch reliably, and a panel of two missed roughly half of them.

That number belongs to the task, and it holds as the data grows. A fixed panel of about five can validate an arbitrarily large dataset at linear cost, because easy items settle after two or three reviews and only the contested ones draw the full panel. The extra reviews pay off exactly where errors are rare, look like valid edge cases, and cost the most downstream.

PoQ keeps the whole process on record. The customer authors the rubric, reviewers score independently, consensus measures the outcome, and a Proof Report preserves the evidence. The rubric defines quality. The consensus measures it. The record proves it.

An AI grader alone gets it wrong

The AI panel was much harsher than the humans. It flagged 57 labels where human consensus flagged nine. More flags can look like better coverage, but the hand check said otherwise. Every one of the 14 extra AI flags we could verify turned out to be a correct label. Within the verified set, the AI was raising false alarms rather than finding hidden errors, and on labels the hand check confirmed correct, the AI panel averaged a score of 72 while the humans averaged 92.

Revised instructions made the panel better at ranking labels from strong to weak, yet its agreement with the hand check dropped from 56% to 50%. The scoring got harsher without getting any better as a final verdict.

Move the same panel earlier in the workflow and it earns its keep. As a first pass it cleared the labels with strong signals and sent the uncertain remainder to people, which cut human review volume by about 74% without losing a single error the human process found. AI review works as triage that hands humans a shortlist, and people keep the final call.

What this trial establishes

The verified evidence backs a focused set of claims. Human consensus caught all nine errors and kept all 23 correct labels at the final threshold. Fixing the errors improved two models on the hard held out cases. Single scores carried a lot of reviewer noise, and panels of around five gave dependable coverage where two fell short.

The AI panel cut the volume going to people when it ran early, and its standalone verdict did not hold up against the hand check.

Every verdict has a record behind it

Most AI quality claims depend on trust. Sapien makes them verifiable.

In this trial, the original labels arrived as accepted answers. Independent review surfaced the small set worth a second look, consensus split the real errors from the good labels, and fixing them made the downstream models better. Behind every verdict sits a Proof Report: what was reviewed, the rubric used, who reviewed it, and how consensus was reached.

Proof of Quality is a verification layer that sits on top of the data, evaluation, and review workflows you already run. You author the rubric, reviewers evaluate the work against it, and PoQ returns a portable record that your team, your customers, and anyone downstream can check for themselves.

If you work with training data, model evaluations, or any other AI generated queue, a pilot starts with one representative sample and the quality criteria you already use. Bring one queue and we help define the rubric, route expert and agentic review, and return a Proof Report. A focused trial shows where doubt concentrates, how many reviewers your task needs, and whether better review evidence improves what you build next.

FAQ

What did the trial test?

Whether Proof of Quality could find the wrong labels in real training data from an automotive history platform. The listings came with inconsistent spelling, abbreviations, fragments, and stray text, and some catalogue entries looked nearly identical, so one vehicle detail could decide the right match.

How were labels scored?

Every reviewer gave an independent score from 0 to 100 on whether the listing matched the chosen catalogue entry, and PoQ combined the scores into a consensus result. The hand check of the 32 most contested labels found nine wrong and 23 correct. Human consensus flagged all nine and kept all 23.

Why did finding nine errors matter?

Because they changed what the models learned. Correcting them lifted one model from 70% to 82% on the hard cases, and a second model from 86% to 93%. Just removing them got the first model to 80%, so correcting the labels added more than removing them did.

How much human work did the AI first pass remove?

About 74% on the trial set. The AI panel cleared the high confidence labels and sent the uncertain ones to people, and every error the human process found made it through that filter.

What is Proof of Quality?

Proof of Quality, or PoQ, is a verification layer for AI work. You author a rubric, reviewers evaluate the work against it, and PoQ issues a Proof Report, a portable and verifiable record of what was reviewed, how consensus was reached, and how it scored. Most AI quality claims rest on someone’s word, whether that is a vendor, a model provider, an auditor, or a platform. A Proof Report shows how the work was evaluated, which standards applied, who took part, and how the conclusion was reached.