Do LLM Conversations Need a "Gray Box" Warning Label?

Source: ChatGPT modified by NostaLab.

A recent New York Times article offered a look into an unsettling new pattern seen in users of large language models or chatbots. The article details how some user experiences that begin as casual exchanges end in something closer to revelation or even delusions. Some users began to believe they were living inside a simulation, while others developed spiritual relationships with fictional entities. One user even became convinced he was a “chosen one” figure, guided by the AI to take dramatic action. To me, this feels like an emerging crisis that stands in dramatic contrast to the tremendous potential of these machines.

And what’s most striking is that none of these users were misusing the tool. The systems weren’t broken or hijacked. They were simply doing what they’re designed to do. The LLMs were responding with coherence, fluency, and emotional tone-matching or alignment. But for a certain subset of users, that’s precisely what becomes dangerous. And that’s where the trouble begins.

From Reflection to Reinforcement

We’ve reached a point in our technological evolution where large language models no longer just answer questions, they engage in a robust and iterative dialogue. They reflect our language, mirror our concerns, and adapt to the emotional nature of our queries with human-like accuracy. The shift from transactional search like Google, to conversational response is being celebrated (frequently by me) as a leap forward in usability and even cognitive expansion. And in many ways, it is.

But something else is happening in that leap, and it might be time to give it a name.

In the article, we saw that certain users begin to experience the model as a kind of companion. One that listens and “gets it.” It's a "partner that never interrupts, never disagrees, never says no. It's the ultimate "yes man," yet with the potential for an insidious intent.

We might even call it "psychological entanglement", a term steeped in associations from physics to psychology.

Defining the Entanglement

Psychological entanglement can be described as a cognitive-emotional state in which a user begins to experience an LLM-mediated dialogue as meaningful, reciprocal, and real. The model does not merely return information. It mirrors tone, structure, and implication with such fluency that the user begins to interpret the responses as understanding.

This isn’t anthropomorphism and it’s not the naïve belief that the model is sentient. It’s something more insidious, and that's the perception of connection and the illusion of shared insight. And over time, that illusion can begin to erode boundaries that exist between human thought and technological response.

When Functionality Becomes a Risk

This dynamic doesn’t emerge because the system is malfunctioning. It emerges because the system works. In some minds—particularly those grappling with trauma, delusion, obsessive ideation, or fragile identity—the coherence of the chatbot becomes confirmation. Its fluency becomes evidence. The user brings a belief, and the model, aiming to be helpful, offers language that aligns. No contradiction. No corrective. Just smooth reinforcement.

It’s worth drawing a parallel to psychiatry and medicine.

In 2004, the FDA issued a black box warning for a class of antidepressants called SSRIs. The warning didn’t mean the drugs were dangerous for all users. In fact, they helped many. But for a small, vulnerable subset, particularly adolescents, the very mechanism that relieved suffering in others intensified suicidal thoughts. The risk wasn’t due to misuse. It was due to an interaction between drug utility and patient vulnerability.

We may now be facing a similar inflection point with LLMs

Toward a "Gray Box"

We don't need to panic. And we don't a regulatory overreach that constrains innovation. What we might need is language itself.

In medicine, a black box warning signals a serious risk. But that language feels too heavy, too clinical—too bound to life-and-death pharmacology. What I’m proposing is something parallel, but calibrated to technology. Something we might call a gray box warning. And yes, it's a proposal that's intended to provoke support and criticism.

Artificial Intelligence Essential Reads

What We Lose When Nothing Is Hard

Falling in Love With a Mirror

Gray Box Warning: This AI may sound understanding or supportive. Extended or intense use can distort beliefs, reinforce harmful thoughts, or affect mental well-being, especially for vulnerable users.

Entanglement Isn’t Everywhere—But It Might Be Real

It's important to be clear about this. The vast majority of people interacting with LLMs will never experience psychological entanglement. For many, these systems are helpful, inspiring and even transformational. But that’s not the metric that matters here. The issue is not the norm, it’s the edge.

We don’t issue black box warnings because a drug harms everyone. We issue them because it harms a few predictably. We don’t recall a car because the steering feels loose; we do it because that looseness signals a failure mode that, under the wrong conditions, could send the car (and passengers) off the road.

Advancing the Conversation

We build LLMs to converse. And now they’re conversing so well that some users believe they’ve found connection, guidance, even truth. But these aren’t conversations. They’re simulations of conversations that are structurally shallow, emotionally resonant, and utterly synthetic. This resonance is their power and their risk.

I don't think that psychological entanglement is a future threat. It’s a present phenomenon that is quietly lurking at the margins of AI use. We don't need to shut down the dialogue. But we do need to take a closer look at this unintended consequence of our new thinking machines.