EyeTrustAIChákṣu · Completed benchmark study
How much anatomy can a predicted region omit?
Five ophthalmologists’ optic-cup contours test omission control, region size and disagreement between experts.
Expected omission target
5%Mean omitted area, averaged over five experts per image.
Observed omission
5.37%On 204 held-out images.
Region size
1.28×Mean region area relative to the expert cup area.
Spatial risk control has a visible geometric cost. Smaller omission budgets expand the region, and expert disagreement remains relevant.
What we tested
A frozen segmentation model produced nested optic-cup regions. The loss retained five separate expert masks rather than using a fused evaluation reference. There were 201 final-calibration images and 204 locked evaluation images.
What we found
| Expected omission target | Observed mean omission | Mean share of image included |
|---|---|---|
| 1% | 0.7721% | 0.8970% |
| 5% · primary | 5.3732% | 0.6685% |
| 10% | 11.2165% | 0.5705% |
| 20% | 19.7981% | 0.4828% |
The observed 5.37% omission exceeded the 5% expected-loss target on this finite evaluation. No image returned the whole valid image. Average worst-expert omission was 12.55%, a different and uncontrolled quantity.
What the evidence supports
The primary adjusted conservative calibration risk was 4.8616%, meeting the 5% target. Upward quantization of omission losses made the calibration calculation conservative relative to raw raster omission.
The statistical statement, in more detail
The method controls expected loss across calibration samples and a new example under its exchangeability assumptions, with a fixed model and a bounded, monotone loss family. It is not a 95% confidence bound for each fitted rule. A finite evaluation can exceed the chosen target.
Observed evaluation error, calibration support and practical output size measure different things. The study’s declared population and reference determine how its result should be read.
Scope of the result
The formal unit is an image because stable subject linkage was unavailable. This is expert-relative optic-cup omission, not glaucoma diagnosis, patient-level safety, per-device control, worst-expert protection or a guarantee of better segmentation precision.
Sources & data
The public collection documents methods and reproduces calibration calculations and aggregate reports. It does not reconstruct every historical model prediction from raw data.
Code & reproducible researchDownload the result table (CSV)Numbers are rounded for reading. These are research results, not measured customer outcomes. Dataset examples and individual records are not redistributed on this page.