Public support enquiries
BANKING77 contains English banking queries with 77 intent labels. After exact-text deduplication: 6,999 training examples, 3,000 calibration examples and 3,072 held-out test examples.
The evidence behind the promise
Inspect the statistical calculation, see what the reply walkthrough illustrates, and understand what we would establish on your own data.
A calculated error bound and held-out results on BANKING77.
Inspect the results ↓02 / Illustrative reply workflowAuthored cases and simulated policy outcomes show the operating flow.
Understand the example ↓03 / Your workflow evaluationEstablish the error, supported bound and review trade-off for your use case.
See the approach ↓Computed results · Support intent classification
This experiment evaluates incorrect intent labels. Factual accuracy of generated replies requires its own evidence and assessment.
One fixed classifier, four error limits. Policies are chosen on calibration data before the test set is evaluated.
| Chosen limit | Calculated upper boundCalibration data | Automatic predictionsTest data | Retained reviewsTest data | Observed errorsAmong automatic predictions |
|---|---|---|---|---|
| 1% | 0.86% | 951 (30.96%) | 2,121 | 4 / 951 (0.42%) |
| 2% | 1.91% | 1,439 (46.84%) | 1,633 | 15 / 1,439 (1.04%) |
| 5% | 4.18% | 2,184 (71.09%) | 888 | 64 / 2,184 (2.93%) |
| 10% | 7.48% | 2,638 (85.87%) | 434 | 178 / 2,638 (6.75%) |
At a 5% limit: the calculated upper bound is 4.18%. On the held-out test enquiries, 2,184 predictions are selected, 888 retain review, and 64 selected predictions are wrong (2.93%). The bound and the test observation are different quantities.
Without selection, all 3,072 predictions are used and 378 are wrong (12.30%). Reviewing every prediction retains 3,072 reviews. These are reference comparisons, not a claim of superiority over a customer’s existing system.
Reviews are counted as routing tasks. This experiment includes no audits, time estimates or assumptions about reviewer accuracy. Production costs and retained work need their own comparison.
Method
BANKING77 contains English banking queries with 77 intent labels. After exact-text deduplication: 6,999 training examples, 3,000 calibration examples and 3,072 held-out test examples.
Word TF-IDF and logistic regression. Maximum class probability ranks predictions; it is not treated as a calibrated probability of correctness.
Exact one-sided binomial upper bounds for ten fixed score thresholds. A Bonferroni correction gives 95% simultaneous confidence under the stated assumptions.
The error is a predicted intent that differs from the dataset label. The target is the error rate conditional on selection. We choose the threshold retaining the most calibration examples among those whose upper bound meets the target.
For k errors among n selected calibration examples, the bound is the 1 − 0.05/10 quantile of Beta(k + 1, n − k). If n = 0 or k = n, it is 1. The ten bounds hold simultaneously with at least 95% confidence under independent sampling, so selection among them is accounted for.
The population interpretation assumes independently sampled queries from the same distribution, correct reference labels, and a model and threshold family fixed before calibration. The public dataset does not establish those conditions for any customer. Conversation-level identifiers are unavailable; near-duplicates and correlated requests may remain.
The official test set has a balanced intent design. Its aggregate rates describe this benchmark, not the class mix of a deployment. There is no guarantee for each intent, language, future request mix or individual prediction.
Four duplicate training texts, seven test texts overlapping training/calibration, and one duplicate test text were removed using a predeclared normalization rule. No model or threshold settings were tuned after seeing calibration or test results.
| Score threshold | Calibration errors / selected | Simultaneous upper bound |
|---|---|---|
| 0.35 | 157 / 2,551 | 7.48% |
| 0.45 | 112 / 2,333 | 6.06% |
| 0.55 | 64 / 2,087 | 4.18% |
| 0.65 | 42 / 1,852 | 3.32% |
| 0.7 | 32 / 1,716 | 2.88% |
| 0.75 | 22 / 1,558 | 2.38% |
| 0.8 | 14 / 1,396 | 1.91% |
| 0.85 | 10 / 1,156 | 1.84% |
| 0.9 | 1 / 858 | 0.86% |
| 0.95 | 0 / 509 | 1.04% |
Reproduce it
The code downloads a pinned dataset version, writes the protocol, trains the classifier, selects policies and then evaluates the test set. The result files include every candidate bound.
python -m venv .venv source .venv/bin/activate pip install -r requirements.txt python evaluate.py
Run with Python 3.13. No API key is required. Download the code and dependency file into the same folder.
Data: Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, 2020. Released by PolyAI under CC BY 4.0. Our classifier is a separate baseline, not the paper’s model.
Original dataset ↗︎Illustrative workflow · The refund example
The refund walkthrough uses authored conversations, scores and policy outcomes. It shows an unsupported claim retained for review and another original reply sent automatically. Those cases illustrate how a policy operates.
Its 1,000-reply workload scenario is simulated. The 3,000-record assessment scenario does not compute a statistical bound. Neither is a measured customer result or evidence of factual-reply performance.
See the business explanationThe next step · Your own evidence
Start with one reply type, language and delivery channel. Keep your existing controls in the comparison.
Define material wrong or unsupported claims, trusted source records and the exact group of original replies covered by the risk target.
Pair drafts with contemporaneous facts and policies. Review routine cases as well as errors, keep related conversations together and separate development, calibration and evaluation.
Freeze the scorer and candidate policies before calibration. Use a method suited to the selected population, then inspect held-out errors, automation and review work against current controls.
Include audits, integration, latency and operating costs. A supported bound and a worthwhile workflow are both needed before moving to a monitored pilot.