What a Mysterious Chinese Room Can Tell Us About Consciousness

Imagine you were locked inside a room full of drawers that are stacked with papers containing strange and enigmatic symbols. In the centre of the room is a table with a massive instruction manual in plain English that you can easily read.

Although the door is locked, there is a small slit with a brass letterbox flap on it. Through it, you receive messages with the same enigmatic symbols that are in the room. You can find the symbols for each message you receive in the enormous instruction manual, which then tells you exactly which paper to pick from the drawers and send out through the letterbox as a response.

The Chinese Room is one of the most important philosophical thought experiments on consciousness and has influenced how AI and machine sentience is viewed.

Leon Gao | Unsplash

Unbeknownst to the person trapped inside the room, the enigmatic symbols are actually Chinese characters. The person inside has unknowingly held a coherent conversation with people outside simply by following the instruction manual but without understanding anything or even being aware of anything other than messages being passed in and out.

John Searle in 2015: One of the most influential contemporary philosophers of mind but has also become controversial (see footnote 1).

Franks Valli | Wikimedia Commons

This story was conceived by the American philosopher John Searle¹ in 1980 and has become one of the most influential and most cited papers in the cognitive sciences and the philosophy of mind with huge implications for how we see computers, artificial intelligence (AI), and machine sentience (Cole, 2023).

Searle (1980) used this thought experiment to argue that computer programs—which also manipulate symbols according to set rules—do not truly understand language or require any form of consciousness even when giving responses comparable to those of humans.

Is AI Sentient?

A Google engineer made headlines in 2022 by claiming that the AI program he was working on was sentient and alive (Tiku, 2022). The recent advance of language-based AI, like ChatGPT, has made many people interact with it just as they would with real people (see "Why Does ChatGPT Feel So Human?").

It is not surprising then, that many users truly believe that AI has become sentient (Davalos & Lanxon, 2023). However, most experts don’t think that AI is conscious (Davalos & Lanxon, 2023; Pang, 2023a), not least because of the influence of Searle’s Chinese room argument.

Consciousness is a difficult concept that is hard to define and fully understand (see "What is Consciousness?" and "The Many Dimensions of Consciousness"; Pang, 2023b; Pang, 2023c). AI programs like ChatGPT employ large language models (LLM) that use statistical analyses of billions of sentences written by humans to create outputs based on predictive probabilities (Pang, 2023a). In this sense, it is a purely mathematical approach based on a huge amount of data.

While this is a tremendous achievement and a hugely complex task, in its essence, AI follows instructions to create an output based on an input, just like the person stuck in the Chinese room thought experiment. Sentience is not required to have sophisticated outputs or even to pass the Turing test—where a human evaluator cannot tell the difference between communicating with a machine or with another human (Oppy & Dowe, 2021).

There is no evidence that AI is sentient. However, even if it was, it may not be able to communicate directly with us and may not understand its own language model.

Joshua Woroniecki | Unsplash

But there is another more troubling implication of Searle’s thought experiment: There is a conscious human being inside the Chinese room who is completely unaware of the communications going on in Chinese. Although we have no evidence suggesting that AI is conscious, let’s assume for a moment that it was: The conscious part is unlikely to understand its own language model and while sentient, may have no idea about the meaning of its own language-based output—just like the person inside the Chinese room.

If AI was conscious, it may be suffering from a kind of locked-in syndrome (see "The Mysteries of a Mind Locked Inside an Unresponsive Body"; Pang, 2023c). It is not clear if this barrier could ever be overcome.

Another implication of the Chinese room argument is that language production does not necessarily have to be linked to consciousness. This is not just true for machines but also for humans: Not everything people say or do is done consciously.

Objections

Searle’s influential essay has not been without its critics. In fact, it had an extremely hostile reception after its initial publication, with 27 simultaneously published responses that wavered between antagonistic and rude (Searle, 2009). Everyone seemed to agree that the argument was wrong but there was no clear consensus on why it was wrong (Searle, 2009).

Artificial Intelligence Essential Reads

How Leaders Can Use AI to Bring Business Objectives to Life

The Large Language Laboratory: AI as Scientist and Subject

While the initial responses may have been reactionary and emotional, new discussions have appeared constantly over the past four decades since its publication. The most cogent response is that while no individual component inside the room understands Chinese, the system as a whole does (Block, 1981; Cole, 2023). Searle responded that the person could theoretically memorize the instructions and thus, embody the whole system while still not being able to understand Chinese (Cole, 2023). Another possible response is that understanding is fed into the system through the person (or entity) that wrote the instruction manual, which is now detached from the system.

Another objection is that AI is no longer just following instructions but is self-learning (LeCun et al., 2015). Moreover, when AI is embodied as a robot, the system could ground bodily regulation, emotion, and feelings just like humans (Ziemke, 2016). The problem is that we still don’t understand how consciousness works in humans and it is not clear why having a body or a self-learning software would suddenly generate conscious awareness.

Many other replies and counterarguments have been proposed. While still controversial, the Chinese room argument has been and still is hugely influential in the cognitive sciences, AI studies, and the philosophy of mind.