EyeTrustAILIDC-IDRI · Completed benchmark study
Preserve disagreement instead of inventing one truth.
Four radiologists’ nodule assessments remain separate references for calibrated ranges of malignancy ratings.
Expected exclusion target
5%Patient-average fraction of expert ratings left out.
Observed exclusion
2.75%Across 85 patients and 139 nodules.
Mean range size
3.79 / 5Ordinal malignancy ratings retained.
Uncertainty includes disagreement in the reference itself. The returned range can reflect that disagreement without collapsing the experts into a majority label.
What we tested
The experiment kept all four radiologists’ ordinal malignancy ratings for each known nodule. Calibration used 83 patients; the locked evaluation used 85 patients and 139 nodules. The formal loss averaged excluded ratings within a patient before averaging patients.
What we found
| Expected exclusion target | Observed patient-average exclusion | Mean ratings retained |
|---|---|---|
| 5% | 2.7549% | 3.7914 |
| 10% | 8.8578% | 2.9640 |
| 20% | 17.9216% | 2.3022 |
All four expert ratings were included for 122 of 139 nodules. That 87.77% full-panel containment is descriptive; it is not the controlled quantity. Broad-disagreement nodules received ranges 0.63 ratings wider on average than unanimous ones.
What the evidence supports
Primary calibration had an adjusted risk of 97 / 2,016 = 4.8115%, meeting the 5% target. The formal unit was the patient, and the controlled quantity was average expert-rating exclusion.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
The reference is subjective radiologist assessment, not pathology. The study does not establish cancer diagnosis, nodule detection, screening or a guarantee for each reader. The pooled 22 excluded ratings out of 556 (3.96%) has different weighting from the primary 2.75% patient-average loss.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.