EyeTrustAIHAM10000 · Completed benchmark study
Control missed labels without returning every label.
Pathology-confirmed skin lesions test prediction sets that contain several plausible diagnoses instead of forcing one answer.
Expected set-miss target
5%Excluding the recorded pathology label from the set.
Observed misses
28 / 6244.49% on the untouched evaluation lesions.
Mean set size
2.24 / 7Labels returned per lesion.
Risk control produced sets narrower than all seven labels, with the cost of uncertainty visible in the number of answers retained.
What we tested
The experiment used 3,085 pathology-confirmed lesions represented by 5,340 images. All images of a lesion stayed together. A frozen five-model scorer used 1,848 development lesions, 613 calibration lesions and 624 untouched evaluation lesions.
What we found
| Quantity | Calibration | Held-out evaluation |
|---|---|---|
| Lesions | 613 | 624 |
| Reference-label misses | 29 · 4.73% | 28 · 4.49% |
| Adjusted calibration risk | 30 / 614 · 4.89% | Not a test statistic |
| Mean labels returned | — | 2.2436 |
No evaluation lesion received all seven labels. The 4.49% observed miss rate has a descriptive Wilson interval of 3.12–6.41%; that interval is separate from the conformal guarantee.
What the evidence supports
The finite-sample adjusted calibration quantity was 30 / 614 = 4.886%, meeting the 5% target. The rule was frozen before the one-time locked evaluation and was not recalibrated afterward.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
The unit is a lesion, not a patient. The release did not provide patient identifiers adequate for a patient-dependence audit. Findings apply to this pathology-confirmed benchmark and do not establish screening, prospective diagnosis, external-site generalization or clinical deployment safety.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.