EyeTrustAINEISS · Completed benchmark study
More evidence makes calibration more stable.
Consumer-injury records test both a final error target and the stability of policies built with different amounts of calibration data.
Expected set-miss target
5%A miss means the recorded class is absent from the returned set.
Observed set misses
493 / 10,0004.93% on the locked 2025 evaluation.
Single-label outputs
50.50%The remaining records received both possible classes.
Calibration can work with little evidence yet vary greatly from sample to sample. More labels made the operating policy substantially more reproducible.
What we tested
A categorical logistic model trained on 2023 CPSC injury records predicted high- or low-acuity eventual emergency-department disposition. Warm-up used 2024 records. Final calibration used 1,000 records from 2025, with a separate 10,000-record evaluation. Survey weights and narratives were not used.
What we found
| Expected miss target | Observed set misses | Single-label outputs | Mean labels returned |
|---|---|---|---|
| 1% | 1.02% | 21.98% | 1.7802 |
| 2.5% | 2.38% | 34.25% | 1.6575 |
| 5% | 4.93% | 50.50% | 1.4950 |
| 10% | 8.55% | 64.35% | 1.3565 |
Across repeated warm-up runs, the 5th–95th percentile width of the chosen threshold contracted by 90.2% between 20 and 1,000 calibration records. Formal support at 20 records did not imply a stable policy or a universal label requirement.
What the evidence supports
Primary calibration recorded 49 misses among 1,000 records. The finite-sample adjustment was (49 + 1) / (1,000 + 1) = 4.995%, meeting the 5% expected set-miss target.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
This is retrospective disposition prediction using recorded assessment fields, not arrival-time triage. The evaluation’s descriptive 95% Wilson interval was 4.52–5.37%. The guarantee is not nationally weighted, class-specific or a statement of patient safety. The fixed scorer reached its training iteration cap without convergence.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.