EyeTrustAIHumAID · Completed benchmark study
How much review can a calibrated policy remove?
Crisis-information posts test risk control within the part of a stream removed from immediate review.
Chosen risk limit
5%Priority posts among those removed from immediate review.
Observed selected error
45 / 1,3273.39% in the separate 2018 evaluation.
Review reduction
13.88%1,327 of 9,559 evaluation posts.
The same classifier can support a defined risk limit for a selected subset. The cost is the amount of work that still needs review.
What we tested
A frozen classifier ranked crisis-information posts. Seven released HumAID categories defined “priority”. Separate roles contained 4,774 calibration posts and 9,559 evaluation posts from the 2018 event mixture. The policy was frozen before evaluation labels were opened.
What we found
| Risk limit | Removed from review | Priority among removed | Priority retained |
|---|---|---|---|
| 2.5% | 760 / 9,559 · 7.95% | 11 / 760 · 1.45% | 99.82% |
| 5% | 1,327 / 9,559 · 13.88% | 45 / 1,327 · 3.39% | 99.28% |
| 10% | 1,860 / 9,559 · 19.46% | 110 / 1,860 · 5.91% | 98.25% |
Priority retention uses a different denominator: 6,225 of all 6,270 priority posts remained for review. That 99.28% retention is an observed result, not a second certified risk limit.
What happened on new events?
The unchanged source policy removed 16.98% of a 2019 replay, with 1.72% selected error. That empirical result did not transfer its 2018 certificate. A separately calibrated 2019 policy removed 18.10%, with 2.87% selected error. The different cohorts do not provide a matched estimate of the benefit of recalibration.
What the evidence supports
At the primary 5% limit, calibration selected 658 posts with 17 priority errors. The one-sided 95% exact-binomial upper bound was 3.85%. The resulting threshold supported the chosen 5% selected-risk limit under the declared 2018 population assumptions.
The statistical statement, in more detail
Exact-binomial tests and a fixed, predeclared testing sequence account for threshold selection. Each supported budget has its own 95% calibration-sampling statement about error within the selected population. This is not certainty for each output or a simultaneous 95% statement across all budgets. The claim depends on the stated sampling and population assumptions.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
Labels are human-agreed crisis-information categories, not verified incident truth. Results are pooled across events; the Kerala slice had 6.10% selected error. Reposts, shared incidents and changing event mixtures matter to sampling assumptions. These are benchmark results, not customer savings or a validated commercial listening workflow.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.