EyeTrustAICBIS-DDSM · Completed benchmark study
Make the cost of a decisive answer visible.
Known mammographic abnormalities test participant-level prediction sets against pathology references.
Expected set-miss target
5%Excluding the participant’s pathology-based class.
Observed misses
12 / 2854.21% in the held-out evaluation.
Single-label outputs
71 / 28524.91% received one class.
A tighter budget produced fewer misses and more two-label answers. Both quantities are needed to judge the result.
What we tested
A frozen EfficientNet-B0 scored known abnormality crops. A participant was malignant if any accepted abnormality had a malignant pathology label. The study used 455 model-development participants, separate 200/200 warm-up roles, and separate 285/285 final-calibration and evaluation roles.
What we found
| Expected miss target | Observed set misses | Mean labels returned | Single-label outputs |
|---|---|---|---|
| 1% | 0.35% | 1.9544 | 4.56% |
| 2.5% | 2.11% | 1.8632 | 13.68% |
| 5% · primary | 4.21% | 1.7509 | 24.91% |
| 10% | 12.28% | 1.4842 | 51.58% |
More calibration examples reduced policy variability in warm-up. The 5th–95th percentile width of the threshold contracted 86.18% between 20 and 150 participants. This is a study-specific stability finding.
What the evidence supports
The 5% calibration recorded 13 misses among 285 participants. Its adjusted quantity was 14 / 286 = 4.895%, meeting the expected set-miss target.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
An earlier tooling task exposed limited outcome-bearing participant-manifest information. The model and policy were not adapted using it; the result is nevertheless retrospective benchmark evidence, not a pristine prospective-style test. The primary descriptive Wilson interval was 2.42–7.21%. Known-abnormality classification is not mammography screening or lesion detection.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.