EyeTrustAICheXpert · Completed benchmark study
Zero observed misses still has a cost.
A small chest-X-ray study tests reference-label inclusion while making the number of unresolved outputs explicit.
Expected set-miss target
5%Excluding the pleural-effusion reference label.
Observed misses
0 / 60The returned sets all included the expert reference.
Single answers
8 / 6052 studies retained both possible answers.
Zero observed misses is not zero risk. Here it came with limited decisiveness: most studies still had two possible answers.
What we tested
A pretrained DenseNet scorer was fixed for pleural-effusion classification. Two hundred expert-reference studies were split into 80 warm-up, 60 final-calibration and 60 separate evaluation studies. The output was a set containing “present”, “absent”, or both.
What we found
| Quantity | Calibration | Held-out evaluation |
|---|---|---|
| Studies | 60 | 60 |
| Reference-label misses | 2 | 0 |
| Adjusted calibration risk | 3 / 61 · 4.92% | Not a test statistic |
| Single / two-label answers | — | 8 / 52 |
| Mean labels returned | — | 1.8667 |
The descriptive Wilson 95% interval for 0 misses in 60 studies is 0–6.02%. Neither zero observed misses nor the calibration statement establishes correctness for each radiograph.
What the evidence supports
Two calibration misses gave an adjusted quantity of 3 / 61 = 4.918%, meeting the 5% expected set-miss target. This controls reference exclusion across all outputs, including two-label answers, not errors conditional on a single answer.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
This is the archived CheXpert validation, separate from the NIH ChestX-ray14 images used in the public walkthrough. Its guarantee does not transfer to those demo images. Repeated studies may share a patient, and upstream scorer selection on the benchmark has not been independently ruled out. No autonomous reporting or clinical benefit is established.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.