// AI for Businesses

6 signs your AI chatbot is starting to hallucinate (and how to respond)

6 sygnałów, że Twój chatbot AI zaczyna halucynować (i jak reagować): okładka artykułu JSON Crew

On February 14, 2024, a Canadian tribunal (British Columbia Civil Resolution Tribunal) ruled that Air Canada must refund a customer, because the airline’s chatbot had promised a “bereavement fare” policy that did not actually exist. The adjudicator wrote plainly: “Air Canada suggests the chatbot is a separate legal entity that is responsible for its own actions. This is a remarkable submission. The chatbot is part of Air Canada’s website. It should be obvious that the company is responsible for all the information on its website”. It was the first high-profile precedent, but not the last.

A month earlier, the British courier company DPD had to shut down its chatbot in a hurry after a customer goaded it into swearing and calling DPD “the worst delivery firm in the world”. The video went viral within 24 hours. In March 2024, journalists at The Markup revealed that the MyCity chatbot launched by New York City Mayor Eric Adams was advising business owners to break the law: firing pregnant women, taking workers’ tips, refusing to rent to people with housing vouchers.

The common denominator of these three stories: an AI chatbot that made something up that had real consequences. This phenomenon is called hallucination. In this article we show 6 signs that your chatbot may be starting to hallucinate, a financial risk calculator, a self-diagnosis quiz and 6 mechanisms that block hallucinations in practice. Finally, we show when the problem runs deeper and the chatbot needs to be replaced with an AI agent.

What AI hallucinations are (in 3 sentences)

A language model (LLM) generates responses by predicting the most likely next words based on statistics. It does not “know” anything about the world; it imitates patterns from its training data. A hallucination occurs when the model generates a response that is linguistically coherent but false (it invents prices, deadlines, court rulings, promises discounts that do not exist, cites non-existent sources). Important: the model does not “lie” consciously; it simply does what it was designed to do, but without a fact-checking mechanism.

Six signs your chatbot is starting to hallucinate

1. It gives specific numbers, prices or dates that are not in your database

This is the most dangerous signal in e-commerce and B2B sales. A customer asks for the price of a product that the chatbot doesn't have in context, and gets an answer “399 PLN” invented by the model based on the statistical price of similar products. The customer files a complaint when they see a different amount at checkout. In the case of Air Canada, the verdict cost the airline 650 CAD in refunds, plus legal costs. This is not about hundreds of PLN, but about a precedent: a company is responsible for every number its chatbot provides to a customer.

2. It cites “sources”, but without links (or with links that don’t work)

When you ask a chatbot for the legal basis of an answer and it replies “under the 2019 data protection act” without linking to a specific official journal entry, it is a sign it is making up the name. In 2023, American attorney Steven Schwartz cited 6 rulings generated by ChatGPT in a New York court. All of them were fabricated. The court sanctioned the law firm, and the case went around the world as a textbook example of hallucination. A chatbot without integration with a real knowledge base will always risk generating “plausible” citations that nobody ever wrote. You can read more about connecting AI to real sources in What RAG is and why AI makes things up without it.

3. It answers out-of-context questions with high confidence

A well-designed chatbot asked “what is the price of our product XYZ” should answer “I can’t find product XYZ in the database” or “I’m passing your question to a consultant”. If instead it says “product XYZ costs PLN 1,200”, and XYZ does not exist in your catalog at all, the chatbot is hallucinating. The key to fixing it: instructions telling the model to say “I don’t know”, plus RAG providing context only from your documentation, plus a guardrail blocking answers on out-of-scope topics.

4. It “changes its mind”: 2 different answers to the same question

Ask the chatbot the same question 5 times in different windows. If it answers slightly differently in each session and two of the five answers contain different numbers or dates, the model is picking answers at random. That is a sign it has no internal source of truth and is just generating from a probability distribution. A well-built chatbot with grounding answers questions from the knowledge base 100% consistently and says “I don’t know” to everything outside it.

5. It makes promises outside its authority

The chatbot promises a customer a “free return within 60 days”, even though your policy is 14 days. It promises a 20% discount for signing up, even though there is no such promotion. It promises delivery “tomorrow by 5:00 PM″, even though the real lead time is 3-5 business days. This is exactly the Air Canada scenario: the chatbot promised a refund policy that did not exist. The tribunal ruled that the company must honor the chatbot’s promise. Working rule: a chatbot must never generate promises. It can describe existing policies, but every new commitment to a customer must go through a human or be hard-coded.

6. It can’t say “I don’t know”

This is the foundation of everything above. If your chatbot responds to an absurd question (“does your company sell electric whales”) with some kind of answer instead of saying “I can’t find information on that topic”, the model is hallucinating by its very nature. Teaching the model to decline to answer is just as important as teaching it to answer. In practice we do this with a system instruction such as: “Answer only questions about X. To everything else, reply: I can’t find that information, I’m passing you to a consultant”.

Risk calculator: how much hallucinations cost you per month

Hallucinations look like a technical problem, but they are a business problem. The calculator below shows the expected cost of hallucinations in your company based on industry, query volume and the average cost of a single wrong answer.

Kalkulator ryzyka halucynacji

Choose your industry (it suggests default values), set the volume and the avg. cost of one wrong answer. Result: expected hallucination cost per month/year + risk level.

10 000
5%
200
Koszt halucynacji / mc
—
Koszt halucynacji / rok
—
—

The calculator assumes that every hallucination generates a real cost (refund, complaint, legal dispute, lost customer). In practice some hallucinations go unnoticed (the customer gets a wrong answer but does nothing about it), while others are very expensive (like the Air Canada case). The real cost is usually somewhere between the expected value and infinity.

Six mechanisms that block hallucinations

In our practice AI implementations we use the mechanisms below as a package. Each one alone partially reduces hallucinations, but only together do they create an architecture you can get your legal and compliance teams to sign off on.

Mechanizm 1: RAG (Retrieval Augmented Generation)

The most effective single mechanism. Instead of letting the model “remember” anything, you give it context only at query time: “Answer the customer’s question based solely on the documentation excerpts below”. The model does not hallucinate out of thin air, it processes specific paragraphs from your knowledge base. Details in the article What RAG is and why AI makes things up without it.

Mechanism 2: Guardrails (blocking out-of-scope topics)

A layer placed over the chatbot that filters both questions and answers. If a customer asks about something outside your domain, the guardrail intercepts the question and returns “this is outside my scope”. If the model generates an answer containing unauthorized promises (e.g. “I guarantee a 50% discount”), the guardrail blocks it. Popular libraries: NVIDIA NeMo Guardrails, Guardrails AI, Rebuff (for detecting prompt injection as in the DPD case).

Mechanism 3: Source citation

In every answer, the model must indicate which part of the knowledge base the information comes from. If it cannot point to a source, it is hallucinating. Perplexity built an entire product on this mechanism. In a company chatbot it looks like this: under each answer you see “source: document-XYZ, page 12″, with a link to the document. The customer can verify it, and the model knows it cannot make things up, because the source is clickable.

Mechanizm 4: Tool use (function calling) zamiast zgadywania

A key mechanism for scenarios involving numbers and system state. Instead of letting the model “tell the customer the price”, you give it a tool: a function getPrice(productId), which the model must call. The model does not come up with a price “off the top of its head”; it calls the function, gets the real value from the database and quotes it to the customer. The same pattern works for stock levels, delivery dates, order statuses and return policies. This is exactly the mechanism that distinguishes a chatbot from an AI agent.

Mechanism 5: Human-in-the-loop in critical scenarios

Not every answer has to go out without a human. In high-risk scenarios (making promises, legal, medical or financial advice), the chatbot prepares a draft and a human approves it. That does not mean AI makes no sense; it means AI does 90% of the work and a human makes the final “send” click. It reduces the risk of hallucination to zero in these scenarios, while keeping service 5-10x faster. More on deploying AI effectively in regulated industries: AI for lawyers: 5 pitfalls.

Mechanism 6: Model choice and on-premise for sensitive scenarios

Not all models hallucinate equally. Claude 4.7 (Anthropic) statistically hallucinates less than GPT-4o, and Perplexity has a built-in source verification layer. For critical scenarios it is worth considering a private on-premise model, fine-tuned only on your verified documents, where you control everything from the data to the inference parameters. Full comparison: AI Automation with ChatGPT? 6 alternatives for companies in 2026 and Private LLM in your company: when on-premise makes sense.

Mechanism comparison: cost, quality, difficulty

MechanismRedukcja halucynacjiImplementation costDifficultyWhen to use
RAG60-80%PLN 15,000-40,000mediumzawsze, fundament
Guardrails20-40%PLN 5,000-15,000easypubliczne chatboty
Source citation30-50%PLN 3,000-10,000easybazy wiedzy, prawo, medycyna
Tool use70-90%PLN 20,000-60,000mediumceny, stany, statusy, obietnice
Human-in-the-loop90-99%proces + koszt godzineasydoradztwo, umowy, wysokie kwoty
Prywatny model + fine-tuning40-60%PLN 60,000-200,000trudnacompliance, sektor regulowany

Rule of thumb from our deployments: the foundation is RAG + guardrails + source citation (together they reduce hallucinations by 80-90% for PLN 25,000-65,000). Only if you still have critical scenarios do you add tool use and human-in-the-loop. A private model comes in only when GDPR or a client contract requires it, which we described in detail in the article Private LLM in your company.

Check in 7 questions whether your chatbot hallucinates

Answer 7 questions honestly. The result is not a precise diagnosis, but an indicator of whether an audit is worthwhile.

When a chatbot is not enough: moving to an AI agent

A chatbot answers. An AI agent acts. This difference is directly related to hallucinations: an AI agent does not guess the price, it calls the function getPrice(). It does not promise delivery tomorrow, it checks stock levels via the API. It does not quote the return policy from memory, it reads it from the document base at the time of answering.

Three situations where it is worth replacing a chatbot with an AI agent:

  • The customer expects AI to complete a task for them (place an order, check a status, reschedule), not just answer a question.
  • Answers require current data from systems (ERP, CRM, warehouse), not just knowledge from a document base.
  • The risk of hallucination is so high that every promise must actually be verified (finance, healthcare, law, high-value B2B deals).

We show more practical scenarios of moving from a chatbot to an agent in B2B sales automation: 7 processes.

Audit your chatbot before a hallucination costs you your first lawsuit

We come to your company, test the chatbot on 50 real scenarios, measure the hallucination rate and design a defense architecture (RAG, guardrails, tool use). Result: a one-page report with concrete recommendations and a quote for the implementation.

Schedule an AI audit

Frequently Asked Questions

Can hallucinations in AI chatbots be eliminated at all?

Down to zero, no. The statistical nature of language models means there is always a non-zero probability of hallucination. However, it can be reduced to a level that is acceptable for the business (below 1% for regular queries, below 0.1% in critical scenarios with human-in-the-loop). The key is architecture: RAG + guardrails + citations + tool use + monitoring.

Who is legally responsible for a company chatbot’s hallucination?

In practice: the company that deployed the chatbot. The Moffatt v. Air Canada precedent (British Columbia Civil Resolution Tribunal, February 2024) is unambiguous: the tribunal rejected the airline’s argument that “the chatbot is a separate entity”. There is no analogous ruling in the Polish legal system (yet), but the principles of contract law and consumer protection lead to the same conclusion. The customer trusts the information on the company’s website, the chatbot is part of the website, the company is responsible.

Does RAG eliminate hallucinations 100%?

No. RAG provides the model with context, but the model can still generate an answer that does not fully match that context (an “in-context” hallucination). It reduces hallucinations by 60-80%, but a full defense package also requires guardrails, citations, tool use and monitoring. Details in the article What RAG is and why AI makes things up without it.

Which model hallucinates the least?

In hallucination benchmarks (Vectara HHEM, TruthfulQA), Claude 4.7, GPT-4o and Gemini 2.5 Pro lead, practically tied. Perplexity has a built-in source verification layer, so for “search and cite” scenarios it is the safest. Open source models (Llama 3.3, Mistral Large) hallucinate 5-15% more often, but the architecture around them makes up the difference. A comparison of models and ChatGPT alternatives in the article AI Automation with ChatGPT? 6 alternatives for companies in 2026.

How do you test a chatbot for hallucinations?

Our standard approach: a set of 50-100 control questions, each with a defined “correct” answer. We run the chatbot on these questions automatically once a week and compare the answers with the reference answers (a second AI model scores the match). Deviations >10% on the same question in subsequent weeks = regression. In addition, adversarial tests: prompt injection, leading questions, attempts to extract promises (discounts, deadlines).

How much does it cost to implement full protection against hallucinations?

The foundation (RAG + guardrails + source citation) is typically PLN 25,000-65,000 one-off + PLN 500-1,500/month for maintenance. The full package with tool use, monitoring and human-in-the-loop for critical scenarios: PLN 80,000-200,000 one-off + PLN 2,000-5,000/month. A private on-premise model adds GPU infrastructure costs (TCO calculator in the article Private LLM in your company).

Doesn't an old rule-based chatbot (without AI) avoid hallucinations?

It doesn’t hallucinate, but it has other problems: it doesn’t understand language variations, every rule has to be written by hand, and customers quickly see it is a “dumb bot” and switch to a consultant. AI delivers a better customer experience, but introduces hallucination risk that has to be addressed through architecture. Going back to rule-based only makes sense in very simple scenarios (5-10 typical questions).

More resources on AI in business

5 out of 5 (3 votes)
Next step

Did you like this article?

If after reading this you feel your company also has processes worth rebuilding, book a free call. Together we will check where sales are really leaking and what makes sense to implement first.

Schedule a call
// Contact

Schedule a free consultation

Tell us about your project and we will contact you with a specific plan and quote. No obligations, specific value.

contact@jsoncrew.com