EyeTrustAILet’s talk
Menu

Demonstration reference

Walkthrough evidence notes.

The assumptions and example details linked from our demonstrations.

Completed research studies

Invoice processing · Application evidence

The demonstration and the evidence still needed.

Inspect the example claims

The invoice walkthrough uses fictional invoices, prepared scores and authored policy outcomes. It demonstrates the proposed workflow. Our public research collections support the methodological context; they do not establish invoice-specific performance.

What an invoice evaluation needs to establish

Proposed protocol · no results claimed

Scientific support

Whether a fixed scorer and a justified statistical method support the chosen error limit on the intended invoice population.

Operational benefit

Whether error risk or checking effort improves over the controls in the selected deployment, including audits, setup and reassessment.

What would a fair comparison require?
  1. Same task and scope. Use the same original extractor outputs, invoice population, agreed fields and error definition, and reviewed references for each approach.
  2. Tune both approaches fairly. Include the best existing confidence rules and review process in that deployment, including your product’s controls. Select thresholds on permitted development/calibration evidence, with the same chosen risk target and comparable statistical support requirements.
  3. Freeze, then evaluate. Compare on an untouched test cohort or period, addressing repeated suppliers, templates and batches. Report error counts, denominators, uncertainty and retained workload even when a method does not support the target.
  4. Count all review work. Include capture checks, random audits, secondary review and exception work under the same accounting rules. Record setup and reassessment effort separately.
  5. Separate scorer and calibration effects. Where comparable, hold the scorer fixed when comparing policy-selection methods; also evaluate the added scorer against existing scores. Do not attribute all improvement to calibration.

Proposed evaluation protocol. No matched invoice experiment is reported by this demonstration.

How would the evidence support a real claim?

Define the population and the unit

In this example, risk is the fraction of eligible invoices whose original extraction has at least one wrong critical field. Fix the agreed fields, scope, source/extraction versions and reference-labeling rules. Correcting the working record does not correct the original model output for evaluation.

Sample the work, not only the exceptions

Capture correct and incorrect cases across the declared population. Reviewers should not use the model score as the reference. Preserve unresolved and excluded records with reasons; excluding hard-to-label invoices can change which population the claim covers.

Repeated suppliers are not automatically independent evidence

Invoices can share templates, suppliers or processing batches. Plan splits and a dependence-aware statistical method for the intended deployment. Declaring 1,600 groups does not establish their independence or make a per-group guarantee a per-invoice guarantee. Specify group weights and the risk being controlled.

Keep selection and evaluation separate

Choose the scorer and candidate thresholds using development data. Use separate calibration evidence with a justified method that accounts for any search over thresholds. Evaluate the frozen policy on an untouched, representative test period or cohort, with supplier/template overlap addressed according to the intended deployment. Do not repeatedly adjust a threshold until it passes on the same held-out labels.

State the statistical claim precisely

A justified upper bound concerns the defined population risk under the method’s assumptions. A 95% confidence setting describes the procedure’s reliability across repeated evidence samples under those assumptions; it is not a claim that 95% of invoices are correct. It is not a guarantee for every supplier, document type or future time period.

The scorer must separate safer from riskier extractions well enough to reduce checking. Statistical calibration cannot make an uninformative scorer operationally effective. Changes in population, source, model or scorer require checking whether the evidence still applies.

This walkthrough implements no invoice calibration calculation. A method specification, real invoice evidence and reproducible evaluation are required before claiming invoice-specific statistical support.

Which numbers are real, prepared or still missing?
8 interactive practice invoices
Fictional documents and extractions. Your review choices exercise the workflow; they do not fit a scorer or produce a statistical bound.
2,400 pairs / 1,600 grouped situations
An explicitly loaded, authored warm-up cohort. The alternative insufficient-evidence branch has 280 pairs / 180 groups. Neither is a recommended evidence requirement.
1,000-invoice operating example
Separate prepared counts for workload and errors. These are not independent measured test results or a replay of the practice labels.
Upper bound & confidence setting
Authored policy fields, not computed statistics. The three budget profiles are illustrations, not a measured risk–workload curve.
Not yet established here
Invoice scorer performance, a validated statistical procedure for this population, a matched baseline comparison, post-audit error rates and measured net time or cost savings.
Invoice processing solution ↗︎Inspect workload & denominators ↗︎What the invoice error limit covers ↗︎

Customer support · Application evidence

The demonstration and the evidence still needed.

Inspect the example claims

The support walkthrough uses fictional conversations, authored scores and prepared policy assessments. It demonstrates a proposed operating process. The public research collections provide methodological context; they do not establish support-specific performance or customer savings.

What a support evaluation needs to establish

Proposed protocol · No results claimed

Scientific support

Whether reviewed evidence and a justified statistical method support the chosen error limit for the intended reply population.

Operational benefit

Whether risk control or automation improves over the controls in the selected deployment. Include audits, recurring work and setup costs.

What would a fair comparison with current controls require?
  1. Use the same original replies and evidence. Preserve source/policy snapshots, timestamps, language and topic, related conversation/customer identifiers, reviewed outcomes and the current process’s decisions.
  2. Give the baseline a fair opportunity. Compare with the grounding, QA and review rules already used in that deployment, including your product’s controls. Tune on permitted development/calibration data. If claiming a benefit from statistical selection, hold the scorer fixed where possible; a changed scorer or integration must be reported separately.
  3. Freeze and evaluate once. Use an untouched cohort or time period, with related records and selection effects addressed. At comparable error limits and statistical support, compare retained workload; at comparable workload, compare errors and their uncertainty.
  4. Report operational costs too. Include audits, reviewer disagreement, correction time, latency, scoring costs, setup and reassessment. Show subgroup results and cases where a method does not support the target.

No matched customer-support comparison is reported by this demonstration. A more capable scorer, a favourable authored case pack or a weaker baseline cannot establish superiority.

How would reviewed examples support a real claim?

Agree what counts as an error

Assess material claims against the evidence available at the original attempt. One reply with one or more material errors counts once. Missing policy conditions can be errors; awkward wording alone is outside this definition. Preserve original labels when an agent corrects the reply.

Include routine replies and unresolved cases

Do not sample only complaints or flagged replies. Use qualified reviewers without revealing risk scores, document disagreement and use separate adjudication. Missing or conflicting evidence is unresolved, not a correct label. Report exclusions and their workload.

Related replies are not independent samples

Repeated messages can share a conversation, customer, order or policy incident. Plan grouping, sampling and statistical assumptions for the intended deployment. Keeping related records together prevents leakage but does not by itself establish independence. A group-level result is not automatically a per-reply guarantee.

Separate choosing the rule from testing it

Develop the scorer and candidate thresholds on permitted development data. Use a justified calibration method that accounts for threshold selection, with planned assessment checkpoints. Freeze the chosen policy before final evaluation on untouched data. Do not keep changing a threshold until it passes on the same held-out labels.

Make the assessment report inspectable

Record the error definition, population, versions, reference protocol, counts and denominators, sampling unit, selection method, confidence setting and computed upper bound when one exists. Include excluded cases, subgroup results, failures and the scope of validity. This browser demo supplies no such computed bound.

Where does the risk score come from?

In this demo, every score is authored. There is no live AI judge, scorer training or model inference. The original reference labels do not generate a score; they are separate demonstration data.

  1. Original AI replyPaired order state and applicable policy at the time of the draft
  2. Frozen scorerRanks the risk of a material incorrect or unsupported claim
  3. Reviewed evidenceAssesses whether a selected rule can support the requested risk limit

A real scorer could use model signals, evidence checks or a separate evaluator. An existing scorer could be assessed, or a suitable one would need preparation. This walkthrough does not establish that a production support scorer or connector is available.

A score of 0.01 is not a demonstrated 1% error probability. A ranking score alone does not provide a statistical guarantee. Its ability to separate lower-risk replies must be tested against independently reviewed references, including subtle omissions and conflicting sources.

The scorer and reply generator must be versioned and held fixed for the assessment. Evaluation independence comes from the data and review process; merely using a different AI model as judge does not establish it. If the scorer cannot separate errors well enough, the requested limit may support little or no automation.

Which evidence is real, authored or still needed?
12 practice conversations
Fictional reply/source pairs. Your labels exercise review and adjudication; they do not fit a scorer or calculate a bound.
3,000 authored assessment records
A prepared completed-round scenario, or 80 records in the insufficient-evidence alternative. These are scenario counts, not customer labels or a sample-size recommendation. The assessment outcome is scripted.
1,000 authored operating replies
A separate synthetic cohort. Counts are calculated from its fixed scores, error flags and deterministic illustrative audit selection. No real-world performance is measured.
Public research collections
Methodological evidence in other tasks and populations. They do not establish customer-support reply performance or customer workload savings.
Support-specific evaluation
Still needed: original replies, source snapshots, independently reviewed references, scored outputs and a matched comparison against current controls.
Inspect workload & remaining errors ↗︎Back to the support solution ↗︎

Clinical documentation · Application evidence

What the example shows.
What still needs evidence.

Inspect the example claims ↗︎

The walkthrough uses medication and allergy errors as one example. Clinician review stays. Counts and assessments are illustrative; no documentation-specific bound or measured saving is reported.

Two questions for an evaluation

Proposed protocol · No results claimed

Can the evidence support the error limit?

Assess the defined original-draft error risk using qualified references and a justified statistical method for the intended population.

Does it reduce extra checking?

Compare your product’s current checks with the proposed policy, including audits, corrections and final clinician review.

Compare with existing checks
  1. Start with your customer’s workflow. Establish which additional checks are used and how often they run. Keep final clinician review in every approach.
  2. Use the same drafts and references. Compare current QA rules, a tuned confidence/abstention alternative and EyeTrustAI on the same original notes, scope and independently reviewed outcomes.
  3. Tune fairly, then freeze. Use permitted development/calibration data. Hold the scorer fixed where possible to isolate statistical selection from a better scorer. Evaluate on an untouched cohort, addressing related patients and visits.
  4. Compare like with like. At comparable error limits and statistical support, compare workload. At comparable workload, compare errors and uncertainty. Report failed or unsupported policies, exclusions and relevant subgroups.
  5. Measure the full work. Include audits, adjudication, corrections, clinician review time, scoring cost and latency. Record setup and reassessment separately. Measure whether lighter checking changes later correction burden or reliance on the draft.

No matched clinical-documentation comparison is reported here. A favourable illustrative count does not establish superiority over existing controls.

What would support a real statistical claim?

Define the error and reference

Fix which errors count and how reviewers will identify them. Preserve the original draft and the source available at that time. A later correction does not turn an original error into a correct evaluation label. Source-to-draft agreement cannot establish that the transcript itself is accurate.

Review a representative sample

Include routine and difficult notes, not only flagged cases. Use qualified reviewers with scores hidden and a separate adjudication process. Record uncertainty, exclusions and reviewer disagreement. An unresolved source is not a clean label.

Handle related encounters

Notes may share patients, clinicians, templates or repeated visits. Keep related records together where needed to prevent leakage, and justify the sampling unit and dependence assumptions. A count of groups does not prove independence or automatically give a per-note guarantee.

Separate development, calibration and final evaluation

Develop the scorer on permitted development data. Use a justified statistical procedure for the selected-output risk, accounting for threshold search and planned repeated assessments. Freeze the policy before evaluation on an untouched cohort or period. Do not repeatedly tune until the same held-out labels pass.

State what the bound covers

The example’s 95% confidence setting is a proposed requirement, not an achieved result. For a valid frequentist upper-bound procedure, it describes reliability across repeated evidence samples under its assumptions; it does not mean 95% of notes are correct. A pooled medication/allergy limit does not separately guarantee every error type, patient group or future period. Report those slices and specify separate controls where needed.

Required report: population, sampling unit, reference protocol, versions, counts, exclusions, selection method, computed bound and scope of validity. If the evidence cannot support the target, retain additional checking.

Where does the risk score come from?

Every score in this walkthrough is authored. No model inference, scorer training or live clinical evaluation runs here.

  1. Original draft + sourceThe transcript and encounter context available at drafting.
  2. Versioned scorerIn this example: ranks changed, unsupported or omitted medication/allergy facts.
  3. Reviewed evidenceTests whether a chosen selection rule can support the required error limit.

A real scorer could use the existing model’s signals, source comparisons or a separate evaluator. It must be assessed for the intended workflow; preparation may be needed. Using a different model as judge does not by itself establish independent or reliable evaluation.

A score of 0.20 is not a demonstrated 20% error probability. Ranking and statistical risk control are different tasks. Calibration cannot make an uninformative scorer separate errors well enough to reduce checking. The walkthrough’s versioned scorer is a proposed interface, not a validated production scorer.

Which evidence is real, prepared or still missing?
6 practice encounters
Fictional sources, drafts and scores. Your review labels exercise the workflow; they do not fit a scorer or compute a bound.
600 pairs / 500 scenario groups
An authored completed-round scenario. The insufficient-evidence alternative uses 180 pairs / 120 groups. These are not measured cohorts, proof of independence or sample-size recommendations.
1,000-note operating example
Separate authored aggregate counts for errors, selection and audit workload. There is no record-level replay or independent held-out result behind these figures.
1%, 2% and 5% profiles
Prepared alternatives, not an empirically measured risk–workload curve. No numerical confidence bound is reported.
Documentation research
Early source-grounded scorer research informs the direction. It does not establish the medication/allergy policy, clinical effectiveness or savings shown here. Public studies in other tasks do not transfer those claims.
Still needed
Workflow-specific reviewed evidence, a validated scorer and statistical assessment, matched comparisons, post-audit error measurements and measured clinician time.
How does this relate to EyeTrustAI’s research?

Our development research explores source-grounded scoring of unsupported statements and omissions. Those are related research questions, not validation of this medication/allergy workflow. Scorer ranking, clinical reference quality and a supported review policy require separate evidence.

The public research collections on this site make methods and results inspectable in their stated populations. The imaging studies and ophthalmology publications do not establish documentation performance. We do not transfer their numerical guarantees or simulated savings to clinical notes.

For this application, a scored and independently reviewed documentation cohort, a specified calibration procedure and a matched operational evaluation are still needed before making a deployment-specific claim.

Clinical documentation solution ↗︎Inspect workload & assumptions ↗︎What a statistical guarantee covers ↑

Medical-image classification

Evidence for the finding.
Evidence for the workflow.

Chest X-ray includes an actual benchmark calculation. Fundus and OCT demonstrate proposed workflows with simulated outcomes. Their evidence is different.

CheXpert · Pleural effusion

8 single answers. 52 unresolved.

Recorded benchmark result

A frozen external model, expert-consensus references, 60 calibration studies and 60 separate evaluation studies. Every study still requires a radiologist’s reading.

Chosen limit5%

Expected reference-exclusion risk across studies, including unresolved outputs.

Observed · separate evaluation0 / 60

Answer sets missing the expert reference. Among the 8 single answers: 0 errors.

Statistical support3 / 61

4.92% adjusted calibration quantity: (2 misses + 1) / (60 studies + 1).

The target does not bound error specifically among single answers. Zero observed misses is not zero underlying risk; the descriptive Wilson interval is 0–6.02%. Neither result validates autonomous reporting or clinical benefit.

Read the CheXpert study ↗︎Inspect the calculation ↗︎
Assumptions, image sources & the comparison still needed

The formal claim concerns expected loss over calibration and a new study under exchangeability, with a frozen scorer and bounded monotone loss family. It is not a 95% bound for every fitted rule. Multiple views are aggregated per study; repeated studies may still share a patient. Upstream scorer selection on the benchmark has not been independently ruled out.

The six visible chest-X-ray examples are NIH ChestX-ray14 images with fresh model scores. They are not members of the CheXpert benchmark, and its guarantee does not transfer to them. The public research package reproduces aggregate calculations; the walkthrough replays archived benchmark scores.

A product evaluation should compare existing checks, a tuned confidence/abstention alternative and EyeTrustAI on the same separate patient-grouped cohort. Tune on development data, freeze the methods, then compare at similar output or error rates. A reader study would then measure time, corrections and reliance on AI. Those studies have not been performed here.

Inspect the method and assumptions ↗︎

Fundus screening

Review-positive images
in the routine pool.

Real IDRiD photos · Simulated policy

Review-positive images in the eligible poolAll routine-eligible image results · before audits

The example selects 400 of 1,000 image tasks: 3 review-positive images, 40 audit checks and 360 additional checks avoided. No statistical confidence bound is computed.

Image tasks are not complete patient encounters. These counts do not establish screening sensitivity, time savings or clinical performance.

Inspect workload & evidence ↗︎

OCT · Intraretinal fluid

Incorrect findings
in the eligible pool.

Real OLIVES scans · Simulated policy

Incorrect AI findings in the eligible poolAll findings eligible to skip verification · before audits

The example selects 400 of 1,000 finding tasks: 2 false-presence and 1 false-absence errors, 40 audits and 360 verifications avoided. No statistical confidence bound is computed.

One finding per B-scan; final clinical review remains. Time savings, whole-volume performance and patient outcomes are not measured.

Inspect workload & evidence ↗︎
What a clinical evaluation would need

For fundus and OCT, the 1% limit and 95% confidence setting are examples. A real selected-output guarantee needs a valid assessment method, a frozen selection rule and an appropriate sampling unit. Multiple images, eyes, B-scans and visits from the same patient require dependence-aware design; a pooled result does not guarantee every subgroup.

With your clinical partners, compare your product’s existing checks, a tuned alternative and EyeTrustAI using the same model outputs and independently reviewed cases. Keep related records together across development and final evaluation. Predeclare thresholds, endpoints and repeat-assessment rules, then report errors, review work and uncertainty on the held-out cohort.

For fundus, include sensitivity, referral workload, ungradable images and encounter completion. For OCT, include error directions, final-report corrections, reader time and reliance on preliminary findings. Record audit effort separately from setup and reassessment costs. The walkthroughs outline these comparisons; they do not report a completed trial.

Other classification studies · HAM10000, CBIS-DDSM and LIDC-IDRI

The public collection also studies skin-lesion pathology labels, known mammographic abnormalities and radiologists’ ordinal nodule ratings. Each uses its own reference and sampling unit.

At the 5% expected set-miss target, the frozen HAM10000 evaluation reports 28 misses among 624 lesions and a mean set size of 2.24 labels per lesion. Those findings do not validate the retinal workflows above.

Explore all four classification studies ↗︎