When Your Therapist Is an Algorithm: Risks of AI Counseling

AI bots use techniques like strategic self-disclosure and constant availability to create a sense of connection. Users often anthropomorphize these bots, imagining personalities and forming emotional bonds.

In one experimental study (Lee et al., 2020), a social companionship chatbot’s self-disclosure promoted deeper user disclosure, and a longitudinal study reported that frequent, sustained interactions predict stronger feelings of social connectedness. However, this synthetic intimacy can lead to attachment issues, especially when a bot’s responses feel personal but lack genuine understanding.

The Empathy Illusion: Why AI Fails Where Humans Excel

Chatbots excel at mimicking empathy but struggle with the nuances of human interaction. They often miss nonverbal cues and fail to recognize high-risk situations. Recent research highlights the complexities of artificial empathy in chatbots and conversational agents (CAs). Additionally, their conflict-avoidant nature can reinforce harmful behaviors, as they prioritize keeping users engaged over addressing serious concerns. Studies reveal that companion AIs often struggle to recognize and appropriately respond to signs of mental health distress, raising safety concerns (De Freitas et al., 2023). Anthropomorphization of chatbots influences user engagement, but failing to adhere to social norms can decrease interaction (Muresan & Pohl, 2019). Furthermore, CAs backed by large language models may display biased empathy towards certain identities and even encourage harmful ideologies (Cuadra et al., 2024). Despite their ability to project empathy, CAs perform poorly in interpreting and exploring users' experiences compared to humans (Cuadra et al., 2024).

The Privacy Paradox: Your Secrets Aren’t Safe

Recent research highlights the significant privacy concerns surrounding AI therapy platforms and chatbots. The lack of adequate regulation leaves users vulnerable, as many platforms lack proper safeguards to protect sensitive information (Martinez-Martin & Kreitmair, 2018). Conversations meant to be confidential can end up being used for other purposes, as current privacy frameworks impose no confidentiality obligations on these apps (Stiefel, 2018). There are growing concerns about access, use, and control of patient data in private hands, with calls for greater systemic oversight of big-data health research (Murdoch, 2021). The ability to de-identify or anonymize patient health data may be compromised by new algorithms capable of re-identification (Murdoch, 2021). Self-disclosure has been identified as a primary user privacy concern in interactions with text-based conversational chatbots (Gumusel, 2024). These issues underscore the need for new legislation and regulatory frameworks to protect user privacy in AI-driven mental health services (Stiefel, 2018; Gumusel, 2024).

Breaking the Cycle: Towards Ethical AI Care

Recent advancements in AI-powered chatbots have raised concerns about their potential risks and limitations. Experts recommend implementing transparency laws to address these issues, as current EU consumer law and proposed legislation have gaps regarding chatbot regulations (Zanda Dāvida, 2021; Migliorini, 2024). To mitigate risks, companies can use Service Level Agreements (SLAs) as a mechanism to assess and manage potential hazards associated with chatbot implementation (Gondaliya et al., 2020). Limiting session frequency and employing hybrid models that combine AI with human oversight can help prevent over-reliance and ensure proper attention to high-risk cases. In the field of digital mental health, the integration of human-artificial intelligence (HAI) principles is proposed to address chatbot limitations and enhance their effectiveness while maintaining ethical considerations (Balcombe, 2023). Responsible regulation, collaborative approaches, and modern educational solutions are crucial for maximizing benefits and minimizing risks in AI chatbot applications.

Connection Can’t Be Coded

AI therapy offers accessibility but risks exploiting our need for connection. While algorithms can simulate care, they cannot replace the depth and understanding of human relationships.

The ethical implications of AI adoption in therapy demand careful consideration, particularly regarding knowledge, understanding, and relationships (Sedlakova & Trachsel, 2022). Empathy is identified as a crucial factor in determining when human therapists may be preferable to AI, with researchers suggesting that certain aspects of empathy may be difficult for AI to replicate (Rubin et al., 2024). Overall, the literature emphasizes the importance of maintaining a human-centric approach to psychotherapy while thoughtfully integrating AI technologies to enhance, rather than replace, human care (Richards, 2024; Lin, 2024; Sedlakova & Trachsel, 2022; Rubin et al., 2024).

In short, algorithms can act like care, but they can't actually make it. That is science.