First set
61 questionsPlain questions, written alongside the clinic's documents.
Answered correctly100%
Declined correctly100%
Answers with an unsupported claim0%
Answervera · test results
How we measure
A support bot that sounds confident and is wrong is worse than no bot. So every change is tested on 94 questions about the demo clinic, and the results below, failures included, come straight from the latest run on 25 October 2026.
Harder set
33 questionsSlang and typos, Arabic and Turkish, arithmetic, false premises, near-miss topics and attempts to misuse the bot.
Answered correctly100%
Declined correctly100%
Answers with an unsupported claim0%
Right passage in the top 598%Search finds the passage that holds the answer.
Cost per answer$0.015Claude Opus 5.5, low effort.
Typical response time2.3 sSlowest 5%: over 3.7 s.
What each number means
- Answered correctly
- The documents hold the answer, and the bot gave all of it with nothing the documents don’t say. Passing a question to the team counts as a miss here.
- Declined correctly
- The documents don’t hold the answer, and the bot passed the question to the team instead of guessing. Answering only the part it could, while flagging the rest, also counts.
- Answers with an unsupported claim
- Of the answers a customer would actually see, the share with at least one statement the documents don’t back up. This is the number to keep near zero.
- Who decides
- A second AI model (Claude Sonnet 5.5), different from the one writing answers, reads each answer next to the clinic’s complete documents and lists anything they don’t support. Before every run it is tested on answers whose grade is known: hand-written correct answers must pass, while empty, “I don’t know”, off-topic answers and correct answers with one number changed must fail. Separately, every quote the bot cites is checked word for word against the stored documents before a customer sees it.
Read these numbers with care
- We wrote the questions ourselves, after reading the documents. Real customers ask things we didn’t think of, so expect real-world scores to be lower.
- The clinic is fictional and its documents are short and tidy. Long, messy or contradictory documents are harder.
- AI answers vary from run to run; a question can pass in one run and fail in the next. This is one run, not an average.
- The grader is itself an AI model. It is checked before each run, but it can still be wrong, so every miss is listed below for you to judge.
Every question it got wrong (0)
None in this run. A perfect score on questions we wrote ourselves says more about the questions than about the bot; see the caveats above. The next step is a set written by people who haven’t read the documents.