OCT
12 OLIVES cross-sections.
Source references and real images illustrate finding review. Policy scores and workload outcomes are prepared; there is no calculated bound or clinical validation for this demonstration population.
Evidence & evaluation
Reference inclusion, additional grading and finding verification have different error definitions. Inspect the claim you intend to offer.
What is guaranteed
The guarantee belongs to an agreed task, model and population under statistical assumptions. A percentage over all studies cannot be presented as an error bound for only the selected results.
OCT · Finding verification
Numerator: incorrect eligible AI findings. Denominator: all findings eligible to skip additional verification, before audits. Both false presence and false absence count for the demonstrated binary fluid finding.
The finding may enter a preliminary draft; final clinical review remains. One B-scan and one finding do not complete an examination or establish a patient-level diagnosis.
Open the OCT exampleFundus · Additional grading
Numerator: review-positive images in the eligible group. Denominator: all routine-eligible image results, before audits. In this demonstration, review-positive means DR grade ≥ 2 or macular-edema risk grade ≥ 1.
This selected-group fraction is not screening sensitivity: its denominator is neither all images nor all review-positive images. Existing referral flags, ungradable-input handling and patient-level screening responsibilities remain.
Open the fundus exampleChest X-ray · Reference inclusion
The CheXpert benchmark evaluates whether the output set excludes the expert reference across the defined population, including unresolved sets. It is not an accuracy guarantee conditional on single answers.
At the 5% target, the separate 60-study evaluation contained 8 single outputs, 52 unresolved outputs and no observed reference misses. These observations do not mean zero future risk. Read the calculated statement and assumptions in the benchmark.
The NIH ChestX-ray14 practice images are separate. Their use in the fictional application does not transfer the CheXpert guarantee to that population.
Inspect the calculated benchmark Open the chest X-ray exampleWhat is demonstrated today
OCT
Source references and real images illustrate finding review. Policy scores and workload outcomes are prepared; there is no calculated bound or clinical validation for this demonstration population.
Fundus
Source grades illustrate image-level review. The simulated policy and outcomes are not a calibration study or measured clinical saving.
Chest X-ray
CheXpert supplies the archived calculation. NIH images support the practice workflow; the clinical interface is fictional.
Chest X-ray: NIH ChestX-ray14, provided by the NIH Clinical Center. Wang et al., ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases, CVPR 2017, pp. 3462–3471. Use and attribution. Fundus: IDRiD, Porwal et al., CC BY 4.0. OCT: OLIVES, original dataset CC BY 4.0; author-hosted dataset-viewer JPEGs. Image files are unchanged. Full provenance and source limits are available in each walkthrough.
What your evaluation establishes
01 / Define
Choose the finding/grade, target risk, intended population, unit of analysis and current comparison process.
02 / Prepare
Preserve image/source identity, versions, scores, quality flags, reference decisions and patient/site groupings.
03 / Assess
Compare errors, unresolved outputs, checks, audits, corrections and clinical time. Establish support for the intended claim.
Include a clinical owner and the product or model owner. A validation partner can help establish reference quality and clinical usefulness without being the purchasing organisation.
Evidence you already have
Assess model/scorer versions, reference definitions, image quality, devices and acquisition conditions. Check patient/site dependence, related images and any prior use in model development or policy selection.
A new finding, device, population or gene-ranking task needs its own references and evidence. Existing labels can help after assessment; there is no universal label count, installation time or guaranteed saving.
Measuring the value
OCT: additional verification avoided, audits, retained verification, corrections and final reporting. Fundus: extra grading avoided, audits, referral burden, image-quality resolution and encounter completion. Measure time rather than inferring it from task counts.
Chest X-ray: report single and unresolved outputs, their observed errors, reading time, corrections and usability. The whole-population guarantee alone does not establish savings or single-answer accuracy.
Compare against the customer’s current workflow, including existing rules and automation. Count preparation, integration, ongoing monitoring and reassessment separately.
Integration and operation
EyeTrustAI
Define and assess the intended guarantee, calibrate where supported and compare risk and operating requirements.
Your product team
Own versioning, interfaces, activation, deployment approvals, monitoring and fallbacks. Production connectors are not supplied by the public demos.
Clinical partners
Own reference reviews, disagreement resolution, clinical interpretation, referrals, reporting and follow-up.
Agree checks for model and device changes, acquisition quality and population shifts, plus audit review and suspension responsibilities. An unchanged model does not establish an unchanged population. Audit findings can arrive after the affected work.
Return to the defined grading, verification or unresolved-output path when support is unavailable. Agree how affected prior cases are assessed and what evidence is needed to resume.