AI Hallucinations in Medicine and Mental Health

Source: Jan Vašek / Pixabay

The term “artificial intelligence (AI) hallucinations” refers to a phenomenon in which an AI algorithms “perceive patterns or objects that are nonexistent or imperceptible to human observers, creating outputs that are nonsensical or altogether inaccurate.”¹ Although they can occur in computers that “see” with a camera or “hear” with a microphone so that they bear some resemblance to the kinds of perceptual hallucinations like voice-hearing that occurs in people with psychosis, these days the term is most often applied to wrong answers generated by large language model (LLM) chatbots like ChatGPT or Gemini.

The Fallibility of AI Chatbots

Countless examples of LLM chatbots generating ridiculously wrong and potentially harmful responses to questions have made news headlines in recent years. Back in 2024, Mark Zuckerberg’s Meta AI was giving obviously wrong answers to simple math problems.² Meanwhile, Google AI was providing users diet tips like eating rocks and trying glue as a pizza topping while offering recommendations to clean washing machines by putting bleach and vinegar in it, which combined to form poisonous—and potentially lethal—chlorine gas.³

Earlier this year, the Chicago Sun-Times was outed for using AI to generate a summer reading list of 15 books, 10 of which turned out not to exist.⁴ Elon Musk’s X-based Grok started churning out posts supporting debunked conspiracy theories about “white genocide” in South Africa and Holocaust denial.⁵ And last month, an official report from Robert F. Kennedy Jr’s "Make America Healthy Again" Report made unsubstantiated claims allegedly backed up by references to published research studies that were completely made up thanks to AI.⁶ This same problem—of supplying both erroneous information and fake references—has been previously noted to be pervasive when ChatGPT is used to write articles related to medical topics.⁷

The potential for harm is obvious. Back in 2023, an AI chatbot offered dieting advice for someone struggling with a restrictive eating disorder.⁸ Other AI chatbots purporting to offer psychotherapy have resulted in patient suicides.⁹ And just last week, it was reported that a user who was struggling with addiction and using a “therapy chatbot” for support was told by the AI app to take a “small hit of meth[amphetamine] to get through [the] week.”¹⁰

Due to such concerning reports, the American Psychological Association has labeled AI chatbots used for mental health support “a dangerous trend” when used for mental health and has urged the Federal Trade Commission to regulate the technology and put safeguards in place for responsible development and transparent marketing.⁹

Why Do AI Hallucinations Occur?

To understand why such “hallucinations” happen, it must be recognized that generative LLMs aren’t really intelligent, smart, or truly creative in the way that we typically associate with human cognition or reasoning.¹¹ They’re only algorithms programmed to predict the next word or sequence of words based on statistical probability, where the “goal is to generate plausible content, not to verify its truth.”¹²

If such responses seem human-like, it’s only because they represent output that’s generated based on a massive number of word strings (e.g., sentences) culled from the internet databases that were written by real people at some point. This is essentially the same process as so-called “AI art” that generates images using extant photographs and art created by real-life human beings and that many have claimed amounts to plagiarism.¹³

AI hallucinations, therefore, reflect an inherent limitation of LLM chatbots that aren’t necessarily concerned with accurate responses. Since they scour huge amounts of data to predict their responses, they’re vulnerable to a “garbage-in, garbage out” problem. And that's potentially dangerous when providing answers to questions about medical and mental health.

Misinformation by Any Other Name

Recognizing that they're false statements, and taking into account the perpetuation of stigma surrounding psychosis, some have argued that we stop calling them AI “hallucinations” and instead refer to them as “fabrications,”¹⁴ “confabulations,”^15,16 and even “bullshit” (defined as statements more concerned with the appearance of truth than actual truth).¹⁷ Personally, I think we ought to sidestep all of that anthropomorphizing, and just call AI hallucinations a new form of misinformation.

Artificial Intelligence Essential Reads

You Complete Me? How AI Hijacks the Journey of Thought

Al—Between the Hype and the Happening

Although LLM programmers are trying to reduce AI hallucinations, it has been argued that this may be impossible. Indeed, recent news indicates that the problem may be getting worse rather than better, with some newer “reasoning systems” generating misinformation as much as 51-79% of the time depending on the task.^18,19

In my book, FALSE: How Mistrust, Disinformation, and Motivated Reasoning Make Us Believe Things That Aren’t True, I argue that false beliefs often stem from trust in misinformation. Within the topic of mental health, misinformation already abounds and has been made worse by social media. For example, a recent investigation by The Guardian found that more than half of the top 100 TikTok videos about mental health contained misinformation.¹⁹

Now that we’re in the midst of the AI technology boom, we can expect it to become more and more developed, relied upon, and trusted as a source of medical and mental health information going forward. We’re already at a point where answers curated AI routinely appear at the top of our internet searches, seemingly saving us the trouble of searching for information ourselves or checking to verify its sources. And so, while AI will no doubt be marketed as a new, improved pathway to truth, it might just be one more form of misinformation to contend with.

Of course, AI does feed us reliable information too. It might yet transform medicine for the better and could even end up replacing human healthcare workers one day. But the current phenomenon of AI hallucinations gives us plenty of reason to be both skeptical and cautious about what the future holds. In the meantime, with LLM chatbots not necessarily programmed for accuracy, humans would be smarter to conclude that AI is not yet ready for primetime as a reliable source of health information.

For more on AI and mental health:

• Is AI Really the Cure for Loneliness and Lack of Connection?
• Will AI Therapy Chatbots Replace Human Psychotherapists?
• Can AI Chatbots Worsen Psychosis and Cause Delusions?
• Deification as a Risk Factor for AI-Associated Psychosis