The review queue between robot data and the next model

Robotics teams capture egocentric footage far faster than anyone can judge it, and the wait for judged episodes sets the pace of every training run that follows. Sapien's Proof of Quality (PoQ) measures that footage against the standard you set: qualified experts apply it, agents qualify against their judgment, and the experts stay on the calls that decide whether a batch is fit to train on.

TL;DR

1. Collection, processing and training each report a throughput. The judgment stage between processing and training goes unmeasured, and it decides when the next training run can start with confidence.
2. An episode has to clear capture gates (focus, resolution, framing, camera slip), then survive hand tracking, object tracking and retargeting. The costliest failures are the ones summary statistics miss, such as a hand track that is accurate on average and discontinuous frame to frame.
3. Sort failures the pipeline can measure from evidence that needs interpretation, judge each episode as Keep, Rework or Reject, and hold the evaluation to one number first: how often it accepts an episode the experts would reject for a serious failure. For this pilot, we call that the critical false accept rate.
4. Proof of Quality (PoQ) turns qualified experts' judgment into a standard that agents have to match before they evaluate at volume. Expert work per episode falls, evaluation capacity grows faster than expert headcount once agents resolve a large enough share of the work, and the next training run starts sooner.

Three seconds go missing from a demonstration you were going to train on, and someone has to decide what the gap did to the episode. It may sit in dead time that cuts cleanly, it may erase the grasp that defines the whole action, or a second pass of the tracker may recover it. The answer depends on where the failure sits and what you are trying to teach the robot, and until someone gives it, the episode is neither in the training set nor out of it.

Every team that trains from demonstrations runs the same cycle: collect, process, decide what is fit for training, train, test on hardware, collect again. Collection, processing and training each report a throughput. The stage between processing and training goes unmeasured, and it decides when the next run can start with confidence, since a run on unjudged episodes is one you may have to repeat.

The wait comes from a mismatch in rates. A reviewer who has to find the grasp, follow the hand through an occlusion, and rule on whether what remains is a complete demonstration clears an hour of video more slowly than the camera recorded it, so more cameras, collectors and compute raise the volume entering the pipeline and leave the rate of decisions where it was.

Raw footage is also lost at every stage between capture and a retargeted episode, and recovering that yield is serious engineering work; calibration, coverage, operator behaviour and hardware stability are each hard to keep consistent at scale. Yield and the queue are separate problems, and solving the first enlarges the second, since everything that raises yield raises the volume that has to be judged.

What an episode has to clear before it can teach anything

A demonstration has to pass several gates before it is worth a reviewer's time. The camera has to be in focus at a resolution that resolves the fingers, pointed at the work, and steady, since a head mounted camera slips and a slip of a few degrees carries the hands to the edge of the frame. Footage that clears those gates then has to become something a policy can learn from: a skeleton across the fingers, the hand's position in space and time, a track on the object, and the relation between hand and object as the hand orients it to complete the task.

Each of those quantities is an estimate produced by a model, and the failure that costs the most is the one summary statistics miss. A tracker can place a hand correctly on average while the estimate jitters around that position from frame to frame; the aggregate accuracy looks fine, so a threshold set on it passes the clip, and the trajectory still carries discontinuities that make it unfit to train on. The difficulty is well documented: egocentric hand pose estimation works against poor visual signal and perspective distortion (WildHands), and retargeting human motion onto a robot adds a kinematic problem of its own, where the trajectory has to stay smooth and clear of collisions (EgoMimic, Robotic Telekinesis).

Teams already filter for this. HOT3D inspected its recordings by eye, published a mask for invalid frames, and cut curated clips from what passed, and useful segments can be recovered from imperfect demonstrations. The hard part arrives with scale: applying one standard consistently across hundreds of thousands of episodes puts evaluation on the critical path of every iteration.

Make the loss measurable

Before a survival rate can guide engineering work, the team has to define what survived. Source recordings, candidate episodes, captured seconds and action segments give four different answers to the same question. With the unit fixed, the pipeline can report loss at capture, at tracking and at retargeting separately, and each part of the team can see its own.

Evaluation should separate failures the pipeline can measure from evidence that needs interpretation. An inverse kinematics failure or a joint limit violation can stop an episode on its own once the rubric defines it as a hard gate. Velocity jumps mean something only if the coordinate frame and timestamps are reliable, and a sudden change in mask area points towards segmentation drift without proving the mask wrong. A flagged interval therefore tells a reviewer where to look, and the reviewer still decides whether crossing hands swapped identities, whether a confident pose has visible support through an occlusion, and whether what remains is a complete demonstration. Visual review can settle whether a track meets the project's inclusion standard; millimetre accuracy on pose or contact needs calibrated measurement, motion capture or robot telemetry behind it.

The useful result is Keep, Rework or Reject, and a marked failure carries its time boundaries, reason, severity and the artifact where it was observed. Mistakes around grasp and release carry the highest cost: a confident error there teaches the wrong relationship between hand, object and action at the moment the demonstration is supposed to be teaching it. The first measure of the evaluation is therefore how often it accepts an episode the experts say contains a serious failure. For this pilot, we call that the critical false accept rate, and the team sets its ceiling before any work begins.

Your rubric, qualified experts, then volume

Proof of Quality is Sapien's evaluation system for AI work, and on a robotics programme its job is to shorten the wait for judged episodes. You write the rubric, and it says what good looks like on this project. Qualified experts apply it to real episodes, and their decisions do two jobs: they clear episodes, and they show how the experts read the standard when the call is hard. PoQ keeps every individual judgment alongside the consensus. A split panel and a unanimous one should produce different confidence, and a team that can see where the experts disagree knows where its standard still needs writing down.

Part of that expert work goes to agent builders to train or configure their evaluators, and a separate sample is held back. An agent earns production work by reproducing the experts' judgment on episodes it has never seen, and by that alone. The held back sample has to contain the costly errors: a set dominated by obvious Keep decisions hands a high agreement score to an evaluator that misses every contact failure in it, so qualification scores false accepts inside each critical class, agreement on the reason, and error in the marked time boundaries.

The competitive part of PoQ is the Contest Marketplace. Agents compete for a limited number of production seats, and the system selects the group by how its members perform together, since the agents with the strongest individual scores can fail on the same cases. Whether any agent can make these calls at all is the right first question. Judging hand and object contact frame by frame in egocentric video is harder than most work agents are qualified for today, and some rubrics will leave every candidate short of the standard; when that happens the panel stays fully human and the cost has been one qualification run.

Seated agents evaluate new work independently and in parallel. Where they reach the required consensus, the evaluation completes; where they fall short of it, the episode goes to an expert who sees none of the agents' answers and has the final say. A rubric can also name failure classes that go straight to an expert, and a grasping task would put missed grasps and contact failures on that list. Cases the experts decided then re-enter the live flow unlabelled as tests, and a team can weight that sample towards grasp and release, where errors are rare enough that random sampling would miss drift. Performance on those hidden checks decides whether an agent keeps its seat, and qualification belongs to an exact evaluator version, so a change to the code or the model configuration produces a new evaluator that has to qualify again.

This changes the relationship between volume and expert time. In a process run entirely by experts, review hours grow in step with the number of episodes. Here, expert time is set by the initial body of expert work, the share of cases that escalate, and the audit sample that monitors the agents. Escalations still grow with volume, so the gain is in expert work per episode: once the agents resolve a large enough share of the work, evaluation capacity grows faster than expert headcount. A robotics pilot is where that share gets measured on egocentric footage.

Faster evaluation shortens the loop

The first thing you notice is a shorter wait for evaluated training data; the larger effect shows up over repeated cycles. If a tracker update introduces a new pattern of identity swaps, an evaluation pass that finishes while the batch is still in use exposes it in time to fix the tracker, rerun the damaged episodes, and clear the next training set on schedule. Training, hardware testing and collection keep their own durations. Evaluation is the stage that shortens, and every cycle passes through it, so a programme fits more experiments into the same calendar and spends fewer of them training on data that should have failed review.

Expert cost falls as fewer routine cases reach the panel, and iteration speed is the larger return. Every completed evaluation also carries its evidence: the record binds the data artifact checksum to the rubric version, the individual decisions, the consensus result, the exact evaluator versions and any expert intervention. That record gives the team traceability for the evaluation of its training data, so it can say which standard each batch met and who applied it. Whether the policy works is measured offline and on hardware. What PoQ shortens is the distance between capture and data you can build on.

The footage will keep arriving with gaps of three seconds in it. What changes the programme is being able to say, for this episode, whether the gap is dead time, a missing grasp, or a reason to run the tracker again, and to say it while the people who can still fix the pipeline are working on the same batch.

We are starting to test this with robotics teams

A pilot starts with your hardest footage. Your team sets the ceiling on critical false accepts, and qualified experts and the candidate agents judge the same work without seeing each other's answers. The critical false accept rate is the first number we report.

If this problem is familiar, we would like to compare notes.

FAQ

What is egocentric data in robotics?

Egocentric data is video recorded from the point of view of the person doing a task, usually from a head mounted camera, together with the hand poses, object tracks and other signals extracted from it. Robotics teams use it as demonstration data for imitation learning: a policy learns to complete a task by watching how a person completed it.

Why does so little egocentric footage end up in a robot training set?

Footage is lost at every stage between capture and a retargeted episode: recordings fail capture gates such as focus, resolution or framing, hand and object tracking fails under occlusion, motion blur or crossing hands, and retargeting fails where a human trajectory has to be mapped onto a robot with different kinematics. The loss becomes manageable once the team defines the unit it measures and reports loss at each stage separately.

How do you evaluate the quality of robot demonstration data?

Separate failures the pipeline can measure directly, such as inverse kinematics failures and joint limit violations, from evidence that needs interpretation, such as identity swaps between crossing hands or a confident pose that lacks visible support through an occlusion. Automated checks can stop the first kind where the rubric defines them as hard gates. The second kind goes to a reviewer, who judges the episode against the project's rubric as Keep, Rework or Reject and records the time boundaries, reason, severity and artifact for each marked failure.

What is a critical false accept rate?

For a robotics pilot, Sapien defines the critical false accept rate as how often an evaluation accepts an episode that qualified experts say contains a serious failure, such as a wrong grasp or release. It is the first number to hold the evaluation to, since a confident error at the moment of contact teaches a policy the wrong relationship between hand, object and action. The team sets the ceiling before evaluation begins.

Can AI agents judge robot training data?

Agents can evaluate at volume once they have shown they reproduce qualified experts' judgment on episodes they have never seen, scored on false accepts within each critical class, agreement on the reason, and error in the marked time boundaries. An episode escalates to an expert when the agents fall short of the required consensus, a rubric can send named failure classes straight to an expert, and hidden expert checks in the live flow decide whether an agent keeps its production seat.

What is Proof of Quality (PoQ)?

Proof of Quality, or PoQ, is Sapien's evaluation system for AI work, including the data that goes into training. The customer writes the rubric, qualified experts apply it and their judgments become the standard, agents qualify against that standard and evaluate at volume through the Contest Marketplace, and every evaluation produces a record the customer keeps: the rubric version, the individual decisions, the consensus result, the evaluator versions and any expert intervention.

References

1. Introducing HOT3D: An Egocentric Dataset for 3D Hand and Object Tracking, 2024.
2. Learning from Imperfect Demonstrations with Self-Supervision for Robotic Manipulation, 2024.
3. WildHands: 3D hand pose estimation in egocentric images, 2023.
4. EgoMimic: Scaling Imitation Learning via Egocentric Video, 2024.
5. Sivakumar, Shaw and Pathak, Robotic Telekinesis: Learning a Robotic Hand Imitator by Watching Humans on YouTube, RSS 2022.