AI Hallucination: Why Your Chatbot Is the World's Most Confident Bullshitter

Every week brings another headline about an AI confidently making something up. A chatbot invents a corporate scandal. A lawyer submits six fabricated legal precedents to a federal judge. A $440,000 government consulting report cites sources that don't exist. The tech industry calls this hallucination, but that word, borrowed from psychology, may actually let developers off the hook by framing a software flaw as a quirky human-like trait.

This episode traces the term's origins back to 1986, when "face hallucination" was a positive descriptor for algorithms that enhanced blurry security camera images by synthesizing realistic details. It was a feature, not a bug. By the 2010s, the word had flipped to describe translation models that prioritized linguistic fluency over factual accuracy, and after ChatGPT's release in 2022, it became the dominant framing for AI error. Not everyone accepts that framing. The episode examines philosopher Harry Frankfurt's rigorous definition of "bullshit" — distinct from lying in that the bullshitter is simply indifferent to the truth — and why a paper in the journal Ethics and Information Technology argues that large language models are, technically speaking, the ultimate bullshit engines.

The mechanics explain why. LLMs are next-word prediction machines, not fact-retrieval systems. To avoid sounding like sterile textbooks, developers inject randomness through a technique called top-k sampling, forcing the model to choose from a pool of likely words rather than always picking the single safest option. That randomness directly correlates with more hallucinations. Anthropic's 2025 interpretability research found a specific neural circuit designed to keep the model quiet when it lacks sufficient data — and hallucinations happen when that circuit misfires, triggering a cascaded error where each false word becomes the context for the next, locking the model into doubling down on its own lies.

The real-world damage runs from darkly comic (ChatGPT endorsing churros as surgical instruments, complete with fake citations from a prestigious science journal) to genuinely costly. Air Canada was ordered by a tribunal to honor a bereavement fare policy its chatbot invented. A lawyer was fined and his case dismissed after submitting AI-fabricated case precedents. Nearly half of AI-generated citations submitted by students in a 2024 study were partially or entirely fake.

But the same mechanism that destroys legal briefs won Nobel Prize-winning science. David Baker's lab used deliberate AI hallucination to design 10 million proteins that don't exist in nature, leading to over 100 patents and 20 biotech companies. The Nobel committee called it "imaginative protein creation." The difference, as Caltech professor Anima Anankumar argues, is that scientific models are taught physics — their hallucinations are grounded in real-world constraints and then validated in a lab.

The episode closes on a question that might be unanswerable: if hallucination is just mathematical imagination, can you cure an AI of making things up without destroying its ability to invent anything new?

Source credit: Research for this episode included Wikipedia articles and transcript materials accessed 4/7/2026. Wikipedia text is licensed under CC BY-SA 4.0; content here is summarized/adapted in original wording for commentary and educational use.