EyeTrustAINTSB · Completed benchmark study
What happens when a calibrated policy gets older?
Aviation-event records test an unchanged policy over time and compare three calibration years on the same later events.
Expected set-miss target
5%Overall severity-label exclusion risk.
Matched evaluation
778The same 2024 events for all three policies.
Primary observed miss
5.40%42 of 778 events under the 2024-calibrated policy.
Recalibration changed the risk and decisiveness of a fixed model. The newest calibration did not automatically produce the lowest observed error.
What we tested
A frozen logistic scorer developed on 2008–2020 records predicted high or low injury severity within recorded U.S. single-aircraft events. Policies were calibrated using 2022, 2023 or 2024 evidence and frozen before comparison on the same 778 events from 2024.
What we found
| Calibration year | Calibration events | Observed set misses | Single-label outputs |
|---|---|---|---|
| 2022 | 625 | 50 / 778 · 6.43% | 27.89% |
| 2023 | 1,189 | 36 / 778 · 4.63% | 22.88% |
| 2024 · primary | 389 | 42 / 778 · 5.40% | 24.94% |
The unchanged 2022 policy moved from 5.27% observed miss in 2022 to 6.98% in 2023 and 6.43% in 2024. Calibration age and calibration sample size both differ in the comparison, so it does not isolate a causal age effect.
What the evidence supports
The primary same-year calibration recorded 17 misses among 389 events. Its adjusted quantity was 18 / 390 = 4.615%, meeting the 5% expected set-miss target. Earlier-year transfers supply descriptive evidence, not a formal guarantee under arbitrary time shift.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
These records already describe accidents or incidents. The study does not predict accident occurrence for arbitrary flights or establish flight safety. Under the primary policy, high-severity misses were 10.48% versus 3.52% for low-severity events; the aggregate target is not a per-class guarantee. The primary descriptive Wilson interval was 4.02–7.22%.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.