Computed results · Support intent classification

A tighter limit retains more review.

This experiment evaluates incorrect intent labels. Factual accuracy of generated replies requires its own evidence and assessment.

One fixed classifier, four error limits. Policies are chosen on calibration data before the test set is evaluated.

3,072 held-out support enquiries · errors are incorrect intent labels
Chosen limitCalculated upper boundCalibration dataAutomatic predictionsTest dataRetained reviewsTest dataObserved errorsAmong automatic predictions
1%0.86%951 (30.96%)2,1214 / 951 (0.42%)
2%1.91%1,439 (46.84%)1,63315 / 1,439 (1.04%)
5%4.18%2,184 (71.09%)88864 / 2,184 (2.93%)
10%7.48%2,638 (85.87%)434178 / 2,638 (6.75%)

At a 5% limit: the calculated upper bound is 4.18%. On the held-out test enquiries, 2,184 predictions are selected, 888 retain review, and 64 selected predictions are wrong (2.93%). The bound and the test observation are different quantities.

Compare the same classifier without selection

Without selection, all 3,072 predictions are used and 378 are wrong (12.30%). Reviewing every prediction retains 3,072 reviews. These are reference comparisons, not a claim of superiority over a customer’s existing system.

Reviews are counted as routing tasks. This experiment includes no audits, time estimates or assumptions about reviewer accuracy. Production costs and retained work need their own comparison.

Method

What was calculated, and from what.

Public support enquiries

BANKING77 contains English banking queries with 77 intent labels. After exact-text deduplication: 6,999 training examples, 3,000 calibration examples and 3,072 held-out test examples.

A fixed, transparent model

Word TF-IDF and logistic regression. Maximum class probability ranks predictions; it is not treated as a calibrated probability of correctness.

Selection-aware bounds

Exact one-sided binomial upper bounds for ten fixed score thresholds. A Bonferroni correction gives 95% simultaneous confidence under the stated assumptions.

Sampling, validity and scope

The error is a predicted intent that differs from the dataset label. The target is the error rate conditional on selection. We choose the threshold retaining the most calibration examples among those whose upper bound meets the target.

For k errors among n selected calibration examples, the bound is the 1 − 0.05/10 quantile of Beta(k + 1, n − k). If n = 0 or k = n, it is 1. The ten bounds hold simultaneously with at least 95% confidence under independent sampling, so selection among them is accounted for.

The population interpretation assumes independently sampled queries from the same distribution, correct reference labels, and a model and threshold family fixed before calibration. The public dataset does not establish those conditions for any customer. Conversation-level identifiers are unavailable; near-duplicates and correlated requests may remain.

The official test set has a balanced intent design. Its aggregate rates describe this benchmark, not the class mix of a deployment. There is no guarantee for each intent, language, future request mix or individual prediction.

Four duplicate training texts, seven test texts overlapping training/calibration, and one duplicate test text were removed using a predeclared normalization rule. No model or threshold settings were tuned after seeing calibration or test results.

Inspect all ten candidate thresholds
Score thresholdCalibration errors / selectedSimultaneous upper bound
0.35157 / 2,5517.48%
0.45112 / 2,3336.06%
0.5564 / 2,0874.18%
0.6542 / 1,8523.32%
0.732 / 1,7162.88%
0.7522 / 1,5582.38%
0.814 / 1,3961.91%
0.8510 / 1,1561.84%
0.91 / 8580.86%
0.950 / 5091.04%

Reproduce it

The calculation is inspectable.

The code downloads a pinned dataset version, writes the protocol, trains the classifier, selects policies and then evaluates the test set. The result files include every candidate bound.

Run locally

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python evaluate.py

Run with Python 3.13. No API key is required. Download the code and dependency file into the same folder.

Data: Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, 2020. Released by PolyAI under CC BY 4.0. Our classifier is a separate baseline, not the paper’s model.

Original dataset ↗︎

Illustrative workflow · The refund example

Two cases explain the flow.
Calibration establishes the guarantee.

The refund walkthrough uses authored conversations, scores and policy outcomes. It shows an unsupported claim retained for review and another original reply sent automatically. Those cases illustrate how a policy operates.

Its 1,000-reply workload scenario is simulated. The 3,000-record assessment scenario does not compute a statistical bound. Neither is a measured customer result or evidence of factual-reply performance.

See the business explanation

The next step · Your own evidence

Establish what your replies
can support at your risk limit.

Start with one reply type, language and delivery channel. Keep your existing controls in the comparison.

  1. Agree the error and population.

    Define material wrong or unsupported claims, trusted source records and the exact group of original replies covered by the risk target.

  2. Prepare independent evidence.

    Pair drafts with contemporaneous facts and policies. Review routine cases as well as errors, keep related conversations together and separate development, calibration and evaluation.

  3. Calculate and compare.

    Freeze the scorer and candidate policies before calibration. Use a method suited to the selected population, then inspect held-out errors, automation and review work against current controls.

  4. Decide what is useful.

    Include audits, integration, latency and operating costs. A supported bound and a worthwhile workflow are both needed before moving to a monitored pilot.