Your AI Chatbot Gave a Wrong Answer. Now What?
A customer forwards a wrong answer from your AI chatbot. Here is the first-hour triage, why the fault is usually a document, and how to stop the repeat.
When your AI chatbot gives a customer a wrong answer, do two things in this order: settle it with the customer inside the hour, then find the document the answer came from. For a small business, AI chatbot wrong answers are rarely the AI inventing something. They are the AI faithfully repeating a price list, a policy, or a rate table that nobody updated.
That order matters. Most owners do it backwards. They open the chatbot settings, poke at the AI, and leave the customer waiting with a quote that may or may not be honoured. This post is the other way round: the customer first, the cause second, the prevention third.
A wrong answer from a document-grounded chatbot is a document problem far more often than an AI problem, so treat it like a wrong price on a shop shelf: fix it with the customer in front of you, then fix the shelf. Honour small differences once, correct large ones with a human message on the same thread, and trace every wrong answer back to the file it quoted. Then decide which questions the chatbot should stop answering from documents altogether.
Why does an AI chatbot give wrong answers?
An AI chatbot that answers from your uploaded documents gives wrong answers for four reasons, and hallucination is the least common of them. The four, in the order we would check them:
- A stale document. The chatbot quoted the 2025 rate card because the 2026 one was announced in a staff WhatsApp group and never uploaded.
- Two documents that disagree. The brochure says a deposit is refundable, the terms PDF says it is not. The chatbot picked one.
- A gap the AI filled. Nobody wrote down the answer to "do you do Saturday installation?", so the model produced a plausible one.
- A calculation it was never given the rule for. Per-square-foot pricing with a minimum charge, or a material multiplier, applied without the minimum.
Only the third is what people mean by "hallucination", and on grounded tasks that rate is now small. Vectara's Hallucination Leaderboard measures how often a model invents facts when it is handed a source document to summarise, and the leading models sit in the low single digits (Vectara Hallucination Leaderboard).
That number is the contrarian point of this post. If the AI invents something in roughly one answer out of fifty when it has a document to work from, and your document is wrong in one answer out of ten, the document is the problem ten times more often. Fixing the AI first is optimising the smaller error.
The best-known chatbot ruling backs this up. In Moffatt v. Air Canada (2024 BCCRT 149), Air Canada's website chatbot told a grieving customer that a bereavement discount could be claimed retroactively within 90 days. A different page on the same website said the opposite. The tribunal ordered the airline to pay the difference, and rejected the argument that the chatbot was a separate entity responsible for its own words (American Bar Association). Read the facts again: that was reason two on the list, two sources that disagreed. Not an invention.
What should you do in the first hour after a wrong answer?
Settle the customer's question before you touch the chatbot. The customer does not care why the answer was wrong. They care whether the number they were given is the number they will pay.
Two rules make this quick. If the wrong answer was in the customer's favour and the difference is small, honour it once and say so plainly. Fighting over RM40 costs more in goodwill than RM40. If the difference is large, or the answer promised something you cannot deliver, a human corrects it on the same WhatsApp thread, apologises once, and offers something concrete for the trouble. Do not let the chatbot send the correction. The customer has just learned not to trust it.
Then, and only then, go find the cause. Here is the full sequence.
How to Fix an AI Chatbot That Gave a Customer a Wrong Answer
Which questions should a chatbot never answer from a document?
Anything that changes faster than you update the file. That single test sorts most of it.
Stock levels change daily. This week's promotion ends Sunday. Available appointment slots change every time someone books. A chatbot reading a PDF from three weeks ago will be confidently wrong about all three, and the customer will act on it.
The fix is not a smarter model. It is routing those question types away from documents. Appointment availability should come from a live calendar sync, so the AI offers slots that are actually open. Stock and promo questions should trigger a handoff to a person while the customer is still in the chat. We covered where that handoff line sits in when a WhatsApp chatbot should hand over, and the short version applies here: hand over before the AI has to guess, not after it has guessed wrong.
A customer who sends a photo of a cracked tile, a car dent, or a skin condition and asks "how much to fix this?" is asking a question no document can answer. The AI cannot assess a photo. The right behaviour is to acknowledge the image, say a colleague will look at it, and notify a human. A chatbot that quotes a price from a photo is producing a wrong answer with extra confidence.
This is also where a second contrarian point belongs. A chatbot that says "let me get a colleague to confirm that" on one question in five is a better chatbot than one that answers everything. Every unanswered question becomes a line in your FAQ document. Every confidently wrong one becomes a refund conversation.
Frequently Asked Questions
How do you stop the same wrong answer happening again?
You stop repeats by treating your documents like a shop treats its shelf labels: one label per item, dated, and replaced the day the price changes. Most small businesses upload a folder of files once, at setup, and never open it again. Six months later the chatbot is answering from a museum.
Three habits fix almost all of it. They sit on top of the knowledge base checklist in our WhatsApp AI chatbot guide, which covers what to upload in the first place. This section is about keeping it true afterwards.
One document per topic. Two files that both mention deposits will eventually disagree. Merge them. The same discipline applies to price lists: it is the reason a single maintained catalog beats a typed price list, which we argued in stop typing price lists. One source of truth for humans is also one source of truth for the AI.
Date the file name and delete the old one. A file called rate-card-2026-10.pdf cannot be mistaken for last year's. Do not archive the old version inside the same knowledge base "just in case". The AI will find it.
Review what the chatbot could not answer, weekly. Fifteen minutes. Every escalation is a question your documents do not cover. Write the answer, upload it, move on. This is the feedback loop most setups skip, and it is why the same wrong answer surfaces in month one and again in month four.
| Cause of the wrong answer | What the customer sees | The fix |
|---|---|---|
| Stale document | Last year's price quoted with confidence | Replace the file the same day the price changes; date the file name |
| Two documents disagree | A refundable deposit that is not refundable | One document per topic; delete the older version, do not archive it |
| Gap the AI filled | A plausible answer nobody wrote down | Add the answer to the FAQ file; tell the AI to hand over when unsure |
| Missing calculation rule | Per-sqft price without the minimum charge | Write the rule out in full: rate, minimum, multipliers, what is excluded |
| Fast-changing fact | A slot or promo that no longer exists | Route to live calendar or human handoff, never to a document |
What does this look like in a real business?
Take a four-person renovation firm in Puchong running Facebook Ads for kitchen cabinet work. Their chatbot quotes per-foot-run cabinet pricing from an uploaded rate card, and it has been fine for months.
In September the owner raises the rate by RM40 per foot run because of material costs. He tells his two sales staff in their WhatsApp group. He does not upload a new rate card. For eleven days the chatbot quotes the old rate to every enquiry, correctly, from the only document it has. Then a customer arrives at the showroom with a screenshot.
The owner raised prices in a staff chat but never updated the uploaded rate card. The chatbot quoted the old price to every lead for eleven days, and a customer turned up with a screenshot of the lower figure.
Honoured the quoted price for that one job, replaced the rate card the same afternoon with a dated file, deleted the old one, and moved the minimum-charge rule into the document in full instead of leaving it in the salesperson's head.
The interesting part is the order of operations the owner adopted afterwards. Price changes go into the document first, then the staff group. That is backwards from how every small business naturally works, and it is the whole fix.
This is also exactly how a platform like Raion HUB's AI chatbot is meant to be run. The AI answers from the PDFs, spreadsheets, and rate rules you upload, detects whether the customer wrote in English, Malay, or Mandarin, and a human can step into the same thread at any point. Photos are saved to the conversation for a person to look at, because the AI does not assess them. The chatbot is only ever as current as the last file you gave it, and that is a feature, not a limitation. It means the fix is always in your hands.
Why the AI is the wrong place to look first
There is a reason every vendor article on this topic lists "hallucination" as the headline cause. It sounds like a technology problem, which means it sounds like the vendor's problem to solve. That framing is comfortable and mostly wrong for a small business.
The honest framing is less flattering. The documents are yours. The two files that disagree are yours. The Saturday installation policy nobody wrote down is yours. An AI chatbot did not create those gaps. It exposed them, at scale, to every customer who asked.
That is also why the distinction between a chatbot and an agent matters here. A chatbot answers. An agent acts on the answer: books the slot, updates the CRM field, fires the follow-up. If the answer is wrong, the agent's action is wrong too. We drew that line in AI chatbot vs AI agent, and the practical consequence is simple: document hygiene matters more, not less, as you let the AI do more.
The bottom line
When an AI chatbot gives a customer a wrong answer, settle it with the customer inside the hour, then trace the answer to the document it came from. The cause is a stale file, two files that disagree, an unwritten answer, or a missing calculation rule far more often than the AI inventing something. Keep one dated document per topic, review unanswered questions weekly, and route fast-changing facts to a human or a live calendar instead of a PDF.
