← Back to the workflow

Evidence & evaluation

Match the guarantee
to the clinical task.

Reference inclusion, additional grading and finding verification have different error definitions. Inspect the claim you intend to offer.

What is guaranteed

Define the error and denominator
before comparing results.

The guarantee belongs to an agreed task, model and population under statistical assumptions. A percentage over all studies cannot be presented as an error bound for only the selected results.

OCT · Finding verification

Incorrect findings among those
eligible to skip extra verification.

Numerator: incorrect eligible AI findings. Denominator: all findings eligible to skip additional verification, before audits. Both false presence and false absence count for the demonstrated binary fluid finding.

The finding may enter a preliminary draft; final clinical review remains. One B-scan and one finding do not complete an examination or establish a patient-level diagnosis.

Open the OCT example

Fundus · Additional grading

Images needing review among those
eligible to skip extra grading.

Numerator: review-positive images in the eligible group. Denominator: all routine-eligible image results, before audits. In this demonstration, review-positive means DR grade ≥ 2 or macular-edema risk grade ≥ 1.

This selected-group fraction is not screening sensitivity: its denominator is neither all images nor all review-positive images. Existing referral flags, ungradable-input handling and patient-level screening responsibilities remain.

Open the fundus example

Chest X-ray · Reference inclusion

Answer sets that miss the expert reference
across all in-scope studies.

The CheXpert benchmark evaluates whether the output set excludes the expert reference across the defined population, including unresolved sets. It is not an accuracy guarantee conditional on single answers.

At the 5% target, the separate 60-study evaluation contained 8 single outputs, 52 unresolved outputs and no observed reference misses. These observations do not mean zero future risk. Read the calculated statement and assumptions in the benchmark.

The NIH ChestX-ray14 practice images are separate. Their use in the fictional application does not transfer the CheXpert guarantee to that population.

Inspect the calculated benchmark Open the chest X-ray example

What is demonstrated today

Real research images.
Different levels of evidence.

OCT

12 OLIVES cross-sections.

Source references and real images illustrate finding review. Policy scores and workload outcomes are prepared; there is no calculated bound or clinical validation for this demonstration population.

Fundus

10 IDRiD photographs.

Source grades illustrate image-level review. The simulated policy and outcomes are not a calibration study or measured clinical saving.

Chest X-ray

A historical calculation and separate practice images.

CheXpert supplies the archived calculation. NIH images support the practice workflow; the clinical interface is fictional.

Research image sources

Chest X-ray: NIH ChestX-ray14, provided by the NIH Clinical Center. Wang et al., ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases, CVPR 2017, pp. 3462–3471. Use and attribution. Fundus: IDRiD, Porwal et al., CC BY 4.0. OCT: OLIVES, original dataset CC BY 4.0; author-hosted dataset-viewer JPEGs. Image files are unchanged. Full provenance and source limits are available in each walkthrough.

What your evaluation establishes

Compare the complete workflow
on independent evidence.

  1. 01 / Define

    One task and clinical reference

    Choose the finding/grade, target risk, intended population, unit of analysis and current comparison process.

  2. 02 / Prepare

    Sources, outputs and reviews

    Preserve image/source identity, versions, scores, quality flags, reference decisions and patient/site groupings.

  3. 03 / Assess

    Risk, retained work and usefulness

    Compare errors, unresolved outputs, checks, audits, corrections and clinical time. Establish support for the intended claim.

Include a clinical owner and the product or model owner. A validation partner can help establish reference quality and clinical usefulness without being the purchasing organisation.

Evidence you already have

Reviewed examples need
a scope and independence check.

Assess model/scorer versions, reference definitions, image quality, devices and acquisition conditions. Check patient/site dependence, related images and any prior use in model development or policy selection.

A new finding, device, population or gene-ranking task needs its own references and evidence. Existing labels can help after assessment; there is no universal label count, installation time or guaranteed saving.

Measuring the value

Report the errors and the work
that stays with the clinical team.

OCT: additional verification avoided, audits, retained verification, corrections and final reporting. Fundus: extra grading avoided, audits, referral burden, image-quality resolution and encounter completion. Measure time rather than inferring it from task counts.

Chest X-ray: report single and unresolved outputs, their observed errors, reading time, corrections and usability. The whole-population guarantee alone does not establish savings or single-answer accuracy.

Compare against the customer’s current workflow, including existing rules and automation. Count preparation, integration, ongoing monitoring and reassessment separately.

Integration and operation

Keep product and clinical responsibilities explicit.

EyeTrustAI

Statistical design and evidence

Define and assess the intended guarantee, calibrate where supported and compare risk and operating requirements.

Your product team

Model and workflow integration

Own versioning, interfaces, activation, deployment approvals, monitoring and fallbacks. Production connectors are not supplied by the public demos.

Clinical partners

References and care decisions

Own reference reviews, disagreement resolution, clinical interpretation, referrals, reporting and follow-up.

When conditions change

Agree checks for model and device changes, acquisition quality and population shifts, plus audit review and suspension responsibilities. An unchanged model does not establish an unchanged population. Audit findings can arrive after the affected work.

Return to the defined grading, verification or unresolved-output path when support is unavailable. Agree how affected prior cases are assessed and what evidence is needed to resume.

Discuss an imaging evaluation Discuss a clinical collaboration