EyeTrustAITweetEval · Completed benchmark study
One sentiment model. Different error budgets.
Sentiment classification and a separate negative-signal study test how risk limits change automation and how policies behave after population change.
Chosen limits
2.5–15%The 2.5% budget supported no automatic selection.
Held-out evidence
12,284Official test posts for sentiment classification.
Observed auto-tagging
6.39–43.41%Across the supported 5%, 10% and 15% policies.
A tighter risk requirement can mean little automation, or none. Reporting the unsupported setting is part of the result.
What we tested
A frozen sentiment classifier and risk scorer were assessed using 7,143 final-calibration posts and the separate 12,284-post official test. The reference was the released negative, neutral or positive label. The threshold sequence and all four policies were fixed before test outcomes were used.
What we found
| Error limit | Auto-tagged | Observed selected error | Calibration support |
|---|---|---|---|
| 2.5% | 0 · 0% | Undefined: no selections | No nonzero policy |
| 5% | 785 · 6.39% | 4.08% | Supported |
| 10% | 3,041 · 24.76% | 9.90% | Supported |
| 15% | 5,333 · 43.41% | 15.19% | Supported |
Development automation did not carry over unchanged: the 10% policy selected 35.96% in warm-up and 24.76% in the final test. Annotation effort and workload gains must be measured for the actual population.
A separate study: negative signals after a population change
A source-calibrated 5% routine-pool limit produced 8.37% negative content when transferred unchanged to the official test population. Target-local recalibration, with the model unchanged, produced 2.09% negative content in the routine pool and 9.70% false alerts under the balanced profile. Together the two automated routes covered 42.82% of the target-local evaluation.
What the two-risk result establishes
The balanced profile supported a joint confidence level of at least 95% by allocating 2.5% failure probability to each component. This was a retrospective target-local analysis using 4,000 new calibration labels, not a pristine new test. It does not demonstrate automatic drift detection. A separate stricter profile did not retain a nonzero alert component under the joint allocation.
What the evidence supports
The 5%, 10% and 15% policies had calibration upper bounds of 3.06%, 8.80% and 12.91%. The 2.5% setting did not support a nonzero policy. Observed error at the 15% setting was 15.19% on the test cohort; the page retains that exceedance.
The statistical statement, in more detail
Exact-binomial tests and a fixed, predeclared testing sequence account for threshold selection. Each supported budget has its own 95% calibration-sampling statement about error within the selected population. This is not certainty for each output or a simultaneous 95% statement across all budgets. The claim depends on the stated sampling and population assumptions.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
Sentiment references are annotation labels, not objective reputation harm. There is no multilingual, per-class or live-deployment guarantee. The sentiment experiment and the later negative-signal recalibration have different designs and must not be pooled as one evaluation.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.