EyeTrustAIGoEmotions · Completed benchmark study
What does a stricter error limit cost in automation?
Fine-grained emotion tagging reveals the trade-off between error tolerance, automation and disagreement among annotators.
Chosen limits
5–20%Four separately calibrated error budgets.
Held-out evidence
10,853Comments, separate from 5,426 calibration comments.
Auto-tagging
3.86–35.86%Observed range across the four policies.
Stricter assurance can leave most work for review. A scientifically supported policy is not automatically an operationally attractive one.
What we tested
A frozen RoBERTa model proposed one of 28 emotion labels. A frozen risk scorer used its outputs to rank unsupported tags. An error meant that the proposed tag was absent from the comment’s agreed reference-label set. Four policies were frozen before the final evaluation.
What we found
| Error limit | Auto-tagged | Unsupported tags | Error among auto-tags |
|---|---|---|---|
| 5% | 419 · 3.86% | 10 | 2.39% |
| 10% | 530 · 4.88% | 16 | 3.02% |
| 15% | 2,931 · 27.01% | 414 | 14.12% |
| 20% | 3,892 · 35.86% | 683 | 17.55% |
Automation expanded into more disputed comments at looser limits. At the 15% setting, observed selected error was 3.14% in the high-consensus slice and 29.87% in the low-consensus slice. The pooled guarantee is not a guarantee for each consensus group.
Why raw confidence is not the same claim
A raw confidence cutoff of 0.8 selected 24.75% of evaluation comments with 13.03% observed error. It was a descriptive comparator, not a policy calibrated to the same statistical requirement. These results do not establish superiority over an equally calibrated alternative.
What the evidence supports
All four budgets supported nonzero policies. Their calibration upper bounds were 4.74%, 6.73%, 14.74% and 17.75%, respectively. The statistical support came from calibration, before inspecting the evaluation outcomes.
The statistical statement, in more detail
Exact-binomial tests and a fixed, predeclared testing sequence account for threshold selection. Each supported budget has its own 95% calibration-sampling statement about error within the selected population. This is not certainty for each output or a simultaneous 95% statement across all budgets. The claim depends on the stated sampling and population assumptions.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
The reference is annotator agreement, not psychological truth. There is no per-emotion, subgroup, multilingual or individual-comment guarantee. The study measures selection fractions, not time savings or the cost of ongoing audits.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.