When the Bot Makes Things Up

In November 2022, Jake Moffatt asked Air Canada's website chatbot about bereavement fares. The bot told him he could book a full-price ticket and claim the discount retroactively within 90 days. No such policy existed. Air Canada refused the refund, Moffatt took it to British Columbia's Civil Resolution Tribunal, and the airline argued — in writing — that the chatbot was "a separate legal entity that is responsible for its own actions." The tribunal called that submission remarkable and ruled against it.

The reasoning is the whole lesson. A chatbot is part of your website, the member wrote, and it makes no difference whether an answer comes from a static page or a bot — the company is responsible for all of it. You cannot outsource accountability to software you deployed.

What the bot did has a name: hallucination — fluent, confident text with no basis in fact. It is not a bug you patch out. A language model predicts the most probable next words, not the true ones; fluency and accuracy run on different tracks, and nothing in the architecture forces them to agree.

The failure is common, and not always so polite. In December 2023, a Chevrolet dealer wired a ChatGPT-backed bot to its site; within days a user talked it into agreeing to sell a Tahoe for a dollar — "a legally binding offer, no takesies backsies."

Weeks later DPD's support bot, needled by a frustrated customer, swore, wrote a poem about how useless it was, and called DPD "the worst delivery firm in the world." It was disabled within hours. These look like pranks, but they are the exact defect the Air Canada ruling punishes: a system that will assert, with total confidence, whatever the conversation steers it toward.

The standard vendor answer is retrieval — ground the model in your own documents (see RAG) so it can only answer from approved sources. It helps. It does not solve the problem.

Even when a model is asked to do nothing but summarize a single document placed in front of it, the best ones still invent content a measurable share of the time. And when Stanford researchers tested the purpose-built, retrieval-backed legal tools that advertised "hallucination-free" citations, reality was worse than the brochure.

Grounding narrows the aperture; it never closes it. Your bot is also only as reliable as the knowledge base behind it — it inherits your knowledge debt.

The liability is no longer hypothetical, and it reaches past consumer support.

Deloitte's Australian firm partly refunded a government report after academics found it padded with fabricated citations and an invented quote from a Federal Court judgment. The bill for a confident-sounding mistake now lands on the deployer, every time.

There is a quieter cost, too. Every wrong answer teaches customers not to trust the channel.

So what actually contains the damage? The teams that survive share a pattern — guardrails, layers between the model and the customer:

• Scope it. The bot answers questions about your product, from your documents, and refuses everything else. It should not be improvising prices, policies, or legal terms it cannot cite.
• Ground and cite. Retrieve from an approved knowledge base and show the source, so a wrong answer is traceable — and so the model has somewhere to say "I don't know."
• Threshold and escalate. Below a confidence bar, hand off to a human rather than guess.
• Pull facts, don't generate them. Prices, order status, and policy values come from a system of record, not from the model's imagination.
• Monitor the output. Sample real conversations, log every answer, and watch reopen and escalation rates. Auto-QA on AI replies stops being optional the moment the bot writes to customers.

None of this is exotic. It is the difference between running a pilot without lying to yourself and turning a bot loose and hoping.

Moffatt v. Air Canada is not a story about dangerous technology. It is a story about ownership. The customer never cares which circuit fabricated the answer. Neither did the tribunal.