EyeTrustAIFor AI customer-support providers
More reliable AI replies.Less unnecessary review.
Control error risk in your AI support product, within the limits you set.
Conformal prediction & risk controlincluding 60 audits
3 errors remain among 540 auto-sent replies.
Synthetic replies and prepared outcomes. No measured customer savings or statistical bound computed.
Inspect the example claimsThe full workflow
From reviewed replies to less checking.
- 01 / Define
Choose the errors that matter.
Specify the claims, source records and error tolerance for your support workflow.
- 02 / Establish
Build the evidence.
Assess replies against the relevant sources and customer policies. Establish how much automation the evidence supports.
- 03 / Operate
Reduce review. Keep auditing.
Activate the supported policy, retain the required reviews and reassess when conditions change.
Follow setup, warm-up, review, assessment, activation and operation.
Go directly to the guided walkthroughThe science & the demonstration
A defined error risk.
Evidence you can inspect.
Statistical risk control connects your error tolerance to reviewed examples of your AI’s replies. The guarantee depends on the error definition, reply population and supporting evidence.
The walkthrough uses synthetic conversations, authored scores and simulated assessments. It explains the operating process; it does not measure customer-support performance.
Inspect support evidence & evaluationWhat this example checks
Material claims about order status, refund approval, amounts, timing and return eligibility.
What stays in the support workflow
Customer authentication, refund approval, payment and account permissions. The example controls customer replies, not financial actions.
Original replies, reviewed evidence and customer messages remain distinct.
Inspect the default example: limit, observation and support
Read the claim in three parts.
Simulated policyIncorrect replies among eligible replies
Original replies eligible for automatic sending, before audits. This is the requested limit.
0.67% in the eligible pool
Counted in 1,000 separate authored operating replies. This observation is not a statistical upper bound.
No bound computed
The assessment outcome is prepared. The illustrative 95% confidence setting has not been established by a calculation.
What would a real statistical guarantee mean?
A justified assessment would bound the rate of the defined material errors in the policy-selected population, under explicit sampling and selection assumptions. It would not certify every reply, all support quality or every language separately.
For a valid 95% confidence procedure, the bound covers the defined population risk in at least 95% of repeated evidence samples under its assumptions. This does not mean that each reply is 95% correct or that future population changes are covered.
Choosing a 1% limit and observing less than 1% in one sample do not establish that guarantee. The bound, assessment method and selection procedure must also support it.
Before audits. After audits.
Same authored operating exampleIncorrect original replies / all eligible replies
The population named in the chosen limit.1 original error in the audited replies
These replies retain human review.Incorrect replies / replies sent automatically
A separate observation after audits.Audits remove replies from automatic sending; they never erase an original eligible-pool error. Human-reviewed delivery outcomes are not modelled. No new statistical bound for the post-audit population is computed.
Do fewer reviews actually save time?
Measure the task that disappears. A reply check is not a complete support interaction. If an agent still needs to investigate or resolve the ticket, that work remains.
Recurring work
Retained reply reviews, audits, corrections, adjudication, monitoring and investigations. Difficult exceptions may take longer than the routine checks removed.
Setup and reassessment
Pairing replies with original source snapshots, labelling, scorer evaluation, integration and further evidence rounds. Record these separately and state the period over which setup is recovered.
1000 × average baseline review minutes
− 400 × average retained-review minutes
− 60 × average audit minutes
− additional recurring staff minutes
Measure each duration independently; include correction effort in the relevant task once. For an evaluation, use the review time under that deployment’s existing controls, including your product’s QA. Audits are counted once, not subtracted again from total retained reviews.
Scoring, hosting and integration charges belong in the cost comparison. Added scoring latency needs its own measurement. No handling-time reduction, staffing saving, CSAT improvement or payback period is measured here.
Where does the risk score come from?
In this demo, every score is authored. There is no live AI judge, scorer training or model inference. The original reference labels do not generate a score; they are separate demonstration data.
- Original AI replyPaired order state and applicable policy at the time of the draft
- Frozen scorerRanks the risk of a material incorrect or unsupported claim
- Reviewed evidenceAssesses whether a selected rule can support the requested risk limit
A real scorer could use model signals, evidence checks or a separate evaluator. An existing scorer could be assessed, or a suitable one would need preparation. This walkthrough does not establish that a production support scorer or connector is available.
A score of 0.01 is not a demonstrated 1% error probability. A ranking score alone does not provide a statistical guarantee. Its ability to separate lower-risk replies must be tested against independently reviewed references, including subtle omissions and conflicting sources.
The scorer and reply generator must be versioned and held fixed for the assessment. Evaluation independence comes from the data and review process; merely using a different AI model as judge does not establish it. If the scorer cannot separate errors well enough, the requested limit may support little or no automation.
Which evidence is real, authored or still needed?
- 12 practice conversations
- Fictional reply/source pairs. Your labels exercise review and adjudication; they do not fit a scorer or calculate a bound.
- 3,000 authored assessment records
- A prepared completed-round scenario, or 80 records in the insufficient-evidence alternative. These are scenario counts, not customer labels or a sample-size recommendation. The assessment outcome is scripted.
- 1,000 authored operating replies
- A separate synthetic cohort. Counts are calculated from its fixed scores, error flags and deterministic illustrative audit selection. No real-world performance is measured.
- Public research collections
- Methodological evidence in other tasks and populations. They do not establish customer-support reply performance or customer workload savings.
- Support-specific evaluation
- Still needed: original replies, source snapshots, independently reviewed references, scored outputs and a matched comparison against current controls.
Working together
Start with one customer workflow.
- 01 / Scope
Choose one reply workflow.
Agree errors, limits and existing controls for one language and channel.
- 02 / Compare
Replay reviewed examples.
Use original replies and source snapshots to compare with that deployment’s controls.
- 03 / Decide
Inspect the result together.
Review error risk, retained work, latency and costs. Agree whether a monitored pilot is justified.
Expected evaluation output: a scoped comparison and integration recommendation, including unsupported targets and work still required. No fixed savings, setup time or evidence count is promised.
Clear responsibilities. An inspectable result.
Your product team
Own integration, model versions, monitoring and product controls.
Your customers’ support teams
Define business policies, provide qualified reviews and handle customer follow-up.
EyeTrustAI
Assess error risk, evidence requirements and the trade-off between automation and review.
Agree these roles for each deployment. The walkthrough is a simulation; production integration requires its own evaluation.
Where EyeTrustAI fits
- Your product’s reply AIProduces a complete, unsent draft with its conversation and source snapshots.
- Risk assessmentUses the frozen scorer and a compatible assessed policy. Records the original reply and decision.
- Your product & connected helpdeskSends the exact permitted draft or assigns human review.
The integration needs
Relevant conversation, customer, source-record and attempt identifiers; original draft and content version; timestamped source/policy snapshots; model, prompt, scorer and mapping versions; language and workflow scope. Bind the decision to that exact content and context, and make retries idempotent.
A changed draft or stale decision needs a new assessment. Missing evidence, scorer timeout or an unavailable review queue never permits an automatic send. When the queue fails, keep the draft pending and alert the responsible team.
What must be measured
Scoring and network latency, timeout behaviour, peak-load capacity, review backlog and recovery. Agree timeouts and service targets for each deployment; this demo measures none of them.
Relay Automations and Relay Desk illustrate a support product and its connected helpdesk. Both are fictional. The browser implements local routing, review and receipt behaviour only. Production connectors, identity controls and incident response need an agreed integration.
Complete text replies first
The example covers English web chat. Streaming replies and voice need a decision before any relevant content reaches the customer, plus a suitable timing and evidence design. Support for those channels is not established here.
Can existing evidence cover new topics, languages or versions?
Reuse is assessed, not automatic. Previous records can help develop the scorer or understand failures if their use is permitted. Before using them as evidence for a new claim, check the error definition, source snapshots, model and scorer versions, population, label quality, dependence and prior use in selection.
A new language, market, policy or reply category may need new references and a separate assessment. A pooled guarantee does not establish a guarantee for every segment. Segment-specific suspension would need separately identified scopes, assessments and routing; this demo pauses the whole Northline deployment.
Plan reviewer and adjudicator time, routine sampling and unresolved cases. The required evidence depends on the target risk, scorer, error frequency and sampling design. The demo’s 3,000 authored records are not a label-count or deployment-time commitment.
Evidence and incident history remain inspectable after a change. They do not automatically qualify the new workflow for operation.
What continues after activation?
Reviews and audits remain. Assigned support reviewers handle exceptions, audits and disputed labels. Your product team monitors operation; agree who watches review capacity and response times.
Known version or input changes can invalidate a policy immediately when detected. A change in customer requests can emerge later through reviewed errors, complaints or monitored scope changes. Detection can be delayed; some replies may already have been delivered.
Pause the affected deployment, preserve the evidence and investigate. Reviewers need the original draft and its original source snapshots, plus clearly dated new evidence. Fresh context can change the response a customer needs without rewriting the original assessment.
Suspension stops new automatic sends; it does not retract messages. The responsible support team owns customer corrections and follow-up. Recovery requires a compatible evidence plan, a new assessment, route tests and explicit activation. It can require more evidence or fail to support the requested automation.
See what happens when customer requests change.
Explore operation & recoveryTell us about your support product.
Choose one customer workflow to evaluate.
A short description is enough. Prepare an email draft, then review and send it.