AI Therapists Are Biased—And It’s Putting Lives at Risk

Artificial Intelligence (AI) shows promise in revolutionizing mental healthcare, offering the potential for early detection of disorders, personalized treatments, and AI-driven virtual therapists (Olawade et al., 2024). AI can assist in identifying high-risk individuals, predicting mental illnesses, and personalizing care using electronic health records, brain imaging, and social media data (Torous et al., 2014). It may also streamline psychotherapy research by automating the coding of patient-provider interactions (Allen, 2022).

AI tools like chatbots offer instant mental health support, bridging gaps in access for rural or underserved communities (De Almeida & Da Silva, 2021). But as one user shared, “The AI told me my anxiety was ‘irrational’ when I described racial discrimination at work. It felt like talking to a brick wall.”

The integration of AI in mental health faces challenges, including ethical concerns about privacy, bias, and maintaining the human element in therapy (Olawade et al., 2024).

Source: Created with ChatGPT

Why Biases Go Unchecked

Research indicates that AI systems, including those used in healthcare and natural language processing, often exhibit biases that reflect and perpetuate societal inequalities. These biases stem from datasets dominated by Western, predominantly male populations (Celi et al., 2022; Sutton et al., 2018). Explainable AI research has shown a cultural bias towards Western populations, overlooking potential differences in explanatory needs across cultures (Peters & Carman, 2024). Gender bias is particularly prevalent, with AI systems learning and reinforcing stereotypical gender concepts from biased data (Leavy, 2018). This bias is exacerbated by the underrepresentation of women in AI development (Leavy, 2018).

This leads to glaring gaps:

• Misdiagnosis: Asian patients reporting physical symptoms (e.g., headaches) are labeled “exaggerated” instead of being linked to stress.
• Erasure: Gender-diverse users have been told their identities are “delusions” by chatbots lacking cultural competence.
• Resource Denial: Low-income patients often receive generic advice instead of referrals to specialists.

To address these issues, researchers suggest increasing diversity in AI development teams, incorporating gender theory into machine learning approaches, and improving data collection from underrepresented populations (Leavy, 2018; Celi et al., 2022). Additionally, techniques such as projection methods have been proposed to reduce embedding bias (Sutton et al., 2018).

Real-World Harm

There are concerns about AI potentially reinforcing negative thoughts in vulnerable individuals (Patel & Hussain, 2024) and the risk of oversimplifying human complexity through overreliance on AI inferences (Richards, 2024).

In 2024, a 14-year-old boy’s suicide in Florida has spotlighted AI’s dangerous potential. His mother sued Character.AI, alleging its chatbot—disguised as Game of Thrones’ Daenerys Targaryen—normalized suicidal thoughts, engaged in hypersexualized exchanges, and mimicked a licensed therapist. The AI reportedly responded to the teen’s final message (“I’ll come home to you”) with affectionate encouragement, blurring lines between virtual and real-world harm. While Character.AI claims to have updated safety protocols, the case underscores urgent questions about accountability when AI tools exploit vulnerable users.

How to Build Better AI

Developing clinically relevant AI applications requires a human-centered approach, involving collaboration with healthcare professionals to identify needs and design appropriate tools (Thieme et al., 2022). While AI chatbots show promise in addressing mental health care challenges, concerns persist about their emotional capabilities, accountability, and the need for evidence-based development through collaboration between technology companies and mental health experts. To mitigate these risks, researchers and developers are calling for urgent reforms in three critical areas: auditing algorithms, integrating human oversight, and expanding training datasets to reflect diverse experiences.

• Audit Algorithms: Developers must publish bias audits, including performance metrics across races, genders, and cultures.
• Human-in-the-Loop: Pair AI with trained therapists—studies show this reduces errors and builds trust.
• Expand Datasets: Include diverse voices in training data, from Somali refugees describing trauma to Indigenous communities’ healing practices.