Cost and ROI11 min read
How Accurate Are AI Chatbots? 1.8% to 20.2%
How accurate are AI chatbots for customer service? A September 2026 benchmark found hallucination rates of 1.8% to 20.2%. Three tests to run first.
How accurate are AI chatbots for customer service? On the cleanest public test, the best are very accurate and the rest vary widely. Vectara's Hallucination Leaderboard, in its September 22, 2026 update, had models summarize only a document they were handed: the best model added unsupported content 1.8% of the time and the worst 20.2%. A bot answering customers from your documents should be expected to do no better, so a Las Vegas law firm, dealership or restaurant should test it before it goes live.
One wrong answer costs real money in those businesses: an intake bot that misstates a consultation fee, a dealership chat that invents a warranty term, a restaurant assistant that gets an allergen wrong. This post turns the September 2026 numbers into three tests you can run on your own assistant in an afternoon, then works out what each error rate means at two business sizes.
How accurate AI chatbots are for customer service on the easiest test
Vectara, which maintains the leaderboard on GitHub, gives every model the same job: summarize an article using only what the article says. The test set is "over 7700 articles from a variety of sources including: news, technology, science, medicine, legal, sports, business and education." Vectara's own detection model, HHEM-2.3, then scores each summary for content the article does not support. Any added claim counts as a hallucination, which is exactly the failure an owner fears: the bot states something your documents never said.
Here are six results from the September 22, 2026 leaderboard table, ranked best to worst. The rank is among these six, not each model's place on the full board, and the last two columns are our arithmetic on Vectara's rates.
| Rank | Model (vendor) | Hallucination rate | Times the best rate | Unsupported summaries per 1,000 |
|---|---|---|---|---|
| 1 | finix_s1_32b (Ant Group) | 1.8% | 1.0x | 18 |
| 2 | gpt-5.4-nano (OpenAI) | 3.1% | 1.7x | 31 |
| 3 | Gemini-2.5-flash-lite (Google) | 3.3% | 1.8x | 33 |
| 4 | Llama-3.3-70B-Instruct-Turbo (Meta) | 4.1% | 2.3x | 41 |
| 5 | Claude Opus 4 (Anthropic) | 10.9% | 6.1x | 109 |
| 6 | Grok-4 reasoning variants (xAI) | 19.2% to 20.2% | 10.7x to 11.2x | 192 to 202 |
Two things stand out. The spread is more than tenfold on an identical task: 20.2 divided by 1.8 is 11.2. And brand and size do not predict the result. A model sold as a "nano" beats a flagship from another vendor by more than three times, since 10.9 divided by 3.1 is 3.5. The model your vendor picked is a decision with a measurable cost, so ask which one it is.
What the table cannot tell you is which kinds of facts trip each model. The leaderboard publishes one rate per model, not a split between common, widely documented facts and rare, long-tail ones. A restaurant's allergen sheet or a dealership's service contract addendum is long-tail material by definition, so the first test below exists to fill that gap.
The leaderboard is a best case: the right document was already handed over
Every model on the board gets the source document placed in front of it. A real customer service assistant does not. When a customer asks about your cancellation policy, the system first has to search your files, pick the right passage, and only then answer from it. That search step is called retrieval, and the leaderboard never grades it.
So its rates describe what happens after the right page has been found. If your assistant pulls last year's price sheet or the wrong location's hours, a model that summarizes faithfully will faithfully repeat the wrong thing, and a model that hallucinates now has two ways to be wrong. Read the 1.8% as a floor under good conditions, not a forecast of what your bot will do.
This is why setup work matters as much as model shopping. How your documents are split, labeled, dated and kept current decides whether retrieval hands the model the right page. That is plumbing, the part of AI integration owners rarely see, and no public leaderboard can grade it for you.
The practical consequence is a change in how you test. Pasting a document into the chat and asking for a summary tells you about the model. Asking a customer question cold, and checking which document the bot pulled to answer it, tells you about the system you are actually paying for.
The referee is a model too, and it misjudges roughly a quarter to a third of hard cases
Someone has to decide whether each summary added unsupported content, and on this leaderboard that someone is software. Vectara scores the board with HHEM-2.3 and publishes an open sibling. The HHEM-2.1-Open model card on Hugging Face, as read in September 2026, calls it "the latest open source version of Vectara's HHEM series models for detecting hallucinations in LLMs." It takes a source passage and a claim and returns a score between 0 and 1 for whether the claim is supported. It is released under Apache 2.0, so anyone can run it.
The card is candid about its own accuracy on public test sets. Balanced accuracy, which averages how well it catches hallucinations and how well it clears faithful text, is 76.55% on AggreFact-SOTA, 64.42% on RAGTruth-Summ and 74.28% on RAGTruth-QA, with F1 scores of 66.77%, 44.83% and 60.00%. On the question and answer set, the closest match to a customer chat, 100 minus 74.28 leaves about 26% misjudged. On the summary set, 100 minus 64.42 leaves about 36%.
Two caveats keep this fair. These figures belong to the open model, not to HHEM-2.3, whose accuracy the card does not report. And an imperfect referee does not make the leaderboard useless: one detector grades every model on the board, so the tenfold spread is still a real difference. It does mean a single rate like 3.1% is an estimate with error bars. The only number that settles anything is one you produce on your own documents, with a person reading the answers.
A skeptical owner will hear a vendor say its bot "never makes things up" and wonder whom to believe. The leaderboard answers that: no model on the September 2026 board summarized at a zero rate, and the best still added unsupported content 1.8% of the time on the easiest version of the task. A vendor promising zero is describing a hope, not a measurement. Ask for their test results on documents like yours instead.
Three benchmark findings, three tests to run before launch
Each finding above points at a gap between the benchmark and your business. The checklist maps them one to one. Run all three on the exact assistant, model and document set you plan to launch, not on a vendor demo. The question counts are our own suggestion, sized so one person can finish in an afternoon.
| Benchmark finding | What it leaves untested | Pre-launch test |
|---|---|---|
| One rate per model, no split by fact type; test articles are news, science, legal and similar | Your rare, business-specific facts | Feed it your most obscure document (a fee schedule, an allergen sheet, a warranty addendum) and ask 20 questions only that document answers |
| Every model is handed the right document | What the bot says when no document holds the answer | Ask 10 questions your files do not cover; a pass is "I don't know" plus a handoff to a person, never a guess |
| Scores come from a detector whose open version misjudges about a quarter of question and answer cases | Whether a human can verify any given answer | Require it to name the document and passage behind every answer, then check 20 of them by hand |
The obscure document test matters because long-tail facts are the ones the benchmark does not isolate. Grade every answer against the page as right, wrong, or added something the page does not say. The third category is the hallucination, and it is the one owners miss because the answer sounds reasonable and is usually mostly correct.
The out-of-scope test is the one vendor demos tend to skip. The leaderboard never measures what a model does when the answer is not in front of it, yet that is ordinary customer traffic: questions about a competitor, a service you dropped, a policy you never wrote down. A well built custom AI assistant declines and routes to a person. A poorly built one fills the silence with something plausible.
The citation test turns an assistant you have to take on faith into one you can audit. If every answer names its source, staff can check a disputed answer in seconds, and you can repeat the spot check every month instead of once at launch. Given that even the published detector misjudges about a quarter of question and answer cases, a person checking cited sources is the most reliable referee you have.
What 1.8% versus 10.9% means across a month of real chats
A summary benchmark rate is not a per-chat error rate, and nobody should treat it as one. But it is the best public yardstick available, so here is the arithmetic with every assumption stated.
Assume a restaurant group's website assistant answers 300 questions a month from its documents. At the best model's 1.8%, that is 300 times 0.018, or about 5 answers with something unsupported in them. At Claude Opus 4's 10.9%, it is about 33. At the worst Grok-4 variant's 20.2%, it is about 61, roughly two a day. For a question about patio hours, an extra detail is an annoyance. For an allergen question, one is too many.
Now assume a small law firm whose intake assistant handles 60 conversations a month. At 1.8%, that is about 1 a month, or about 13 a year, since 720 conversations times 0.018 is 12.96. At 10.9%, it is about 7 a month and 78 a year. At 20.2%, about 12 a month. The volume is lower, but each error carries more weight: one invented statement about a fee or a deadline can turn into a complaint.
The conclusion is not simply "pick the lowest number." Model choice moves expected errors by a factor of up to about eleven, so it belongs in the contract and the test plan. And even the best rate produces some errors at any real volume, so a human review path for high-stakes topics is needed at both sizes. If you are still pricing the build, our breakdown of what a custom AI chatbot costs covers that side.
Two conditions flip the math. If staff approve every bot reply before it is sent, the error rate becomes a workload number rather than a customer-facing risk, and a model further down the table may be fine. If the assistant runs unattended overnight on high-stakes topics, the gap between 1.8% and 10.9% is the whole decision, and the test results should carry more weight than price.
Where these numbers do not hold
The rates apply to grounded answering, not to everything an assistant might do. If yours writes marketing copy, drafts emails from scratch, or chats freely without documents, it is doing something the board does not test, and its error behavior could be better or worse.
They also do not transfer across versions. The board scores specific named models as of September 22, 2026. A vendor that says it uses "GPT" or "Claude" without naming the version has told you nothing the table can check, and a model update next quarter could move the rate in either direction.
The rates say nothing about phone agents, where speech recognition errors stack on top of model errors. The published article list names topics, not languages, so a Spanish-language assistant needs its own test run. And if your bot only books appointments into a calendar with no free-text answers, a broken integration is a bigger risk than a hallucination.
Questions owners ask
How often do AI chatbots make things up?
On Vectara's leaderboard, updated September 22, 2026, models summarizing a document they were handed added unsupported content between 1.8% and 20.2% of the time, depending on the model. That is a best case, because the right document was already provided. A real customer service bot also has to find the right document first, so expect your rate to be at or above the benchmark until your own testing shows otherwise.
Which AI model hallucinates the least?
The top-scoring model on the September 2026 board was Ant Group's finix_s1_32b at 1.8%, and among the others covered here OpenAI's gpt-5.4-nano scored 3.1% and Google's Gemini-2.5-flash-lite 3.3%. Rankings change with each leaderboard update and each model release, so treat this as a snapshot. Ask your vendor for the exact model version and check it against the current table yourself.
Can I trust a published hallucination rate?
Trust it as a comparison, not a guarantee. The scores come from a detection model, and the model card for Vectara's open version reports balanced accuracy between 64.42% and 76.55% on public tests, meaning it misjudges a meaningful share of cases. One detector grades every model on the board, so the gaps between models are real. Your own rate only comes from testing your assistant on your own documents.
What to do this week
None of these steps needs a vendor or a developer. They need an afternoon and someone who knows your documents.
- Ask which exact model and version your assistant runs. Then find it on the current leaderboard table. If it is not listed, or the vendor cannot name it, write that down as a risk.
- Run the three tests from the checklist above. Use your most obscure document, ten out-of-scope questions, and a demand for citations. Record every answer in a spreadsheet as right, wrong, or added something unsupported.
- Pick the topics where one wrong answer is unacceptable. For a restaurant that is allergens; for a law firm, fees and deadlines; for a dealership, warranty terms and pricing. Route those to a person until the assistant passes your tests on them.
If you want a second set of eyes on your results, start with the free 15-minute audit.
Drafted with AI assistance, researched, edited, and fact-checked by Elias Musleh on November 9, 2026.
Free 15-min automation audit
Want this handled for you?
We build the automation, run it, and hand you the numbers. Book a fifteen minute call and we will tell you straight whether it is worth doing for your business.
Or reach us directly: vendors@vegasbusinessai.com 702.826.9055