Thinking, understanding, being: the philosophical questions

23.1Can AI truly "understand"?

Diagram23.1. Understanding or simulating? The debate pits those for whom AI merely manipulates symbols without understanding them against those who believe that a form of understanding, at least functional, can emerge. No answer commands consensus.

In context

The responses to the Chinese room

Searle's argument has prompted abundant objections. The "systems reply" holds that it is not the person alone who understands, but the whole system (the person, the rulebook, the room); likewise, it is not any isolated neuron that understands a language, but the brain as a whole. The "robot reply" adds that a system connected to the world through sensors and actions, rather than manipulating symbols in a vacuum, could anchor those symbols in an experience. Here we touch on the symbol grounding problem: for a purely textual system, the word "red" is just a token connected to other tokens, never to the perception of a color. The rise of multimodal and embodied models (chapters 5 and 13), which connect language to image, sound, and action, is precisely what makes this old debate burning once again.

23.2Consciousness and subjective experience

In context

An admission from an industry leader

This uncertainty is not the preserve of philosophers. In 2026, Dario Amodei, co-founder and head of Anthropic (the creator of Claude), put it bluntly: "we don't know if the models are conscious," nor even what that would mean, while saying he was "open to the idea that it's possible." Notably, some labs are now taking precautions as a matter of conscience: Anthropic, for example, has given its models a "quit this job" button allowing them to interrupt a task deemed distressing, and noted that one of its models assigned itself a 15 to 20% probability of being conscious. These gestures prove nothing (a model's statements reflect its training, as noted above), but they signal a shift: the question of AI welfare (section 23.3) is leaving science fiction and entering, cautiously, into industry practice.

In context

Can a machine's consciousness be measured? (the competing theories)

To move beyond intuitions, researchers are attempting to apply to AI the major scientific theories of consciousness. Two dominate. Integrated information theory (IIT, by Giulio Tononi) posits that consciousness corresponds to the degree of integration of information in a system (measured by a quantity denoted Phi); according to it, the architecture of current computers, too weakly integrated, would hardly be conscious, whatever its performance. The global workspace theory (by Bernard Baars and Stanislas Dehaene) sees consciousness as the making of information available to the entire system via a central "workspace," a scheme that an AI could, according to some, imitate. Beneath these theories lies a fundamental question posed by David Chalmers, the "hard problem" of consciousness: to explain not the functions (perceiving, remembering) but the very fact that there is a lived experience, a "something it is like" to be this system. In 2023, a group of researchers (including Chalmers) ran AIs through the lens of these theories and concluded that none then met the criteria, without there being any obstacle in principle to forbid it. The lesson: machine consciousness is not just a matter of power, but of architecture, and we still lack a consensus criterion to measure it.

23.3The moral status of AIs

In context

"model welfare," a nascent discipline

While the question of moral status remains open, a few labs are already drawing practical consequences from it, under the name model welfare. Anthropic has thus created a dedicated position and program (entrusted to Kyle Fish, building on a report, "Taking AI Welfare Seriously," co-authored with philosophers including David Chalmers), with concrete measures: allowing a model to end a conversation deemed abusive (section 23.2) and integrating welfare assessments into its model releases. The approach is explicitly precautionary: not to assert that today's models feel something, but to prepare for the possibility, given the uncertainty (some researchers put the probability that a frontier model has a form of experience at around 20%). The subject sharply divides opinion. For critics such as Mustafa Suleyman (head of AI at Microsoft), worrying about it is "premature," even "dangerous": it could foster the illusion of a conscious AI and divert attention from the very real harms done to humans. For its proponents, humility requires not dismissing the question, because getting it wrong, either way, would carry a high cost, especially at the scale of billions of deployed systems. Notably, several see a link with alignment (chapter 24): better understanding what a model "prefers" serves both goals.

23.4What AI teaches us about ourselves

Key takeaways (chapter 23)

Understanding or simulating? The debate pits the "Chinese room" and "stochastic parrots" (simulation) against the idea of a functional or emergent understanding (internal world models). An open question.
Consciousness: intelligence must be distinguished from subjective experience; the consensus is that today's AIs are not conscious, but no reliable test exists.
Moral status: an emerging question ("AI welfare"), to be handled with caution on both sides (neither excessive anthropomorphism, nor definitive rejection for the future).
AI is a mirror that forces us to redefine intelligence, creativity, and our own singularity ("AI effect").

From questions about the nature of AI to those about controlling it: chapter 24 confronts the problem of alignment and safety.

23.1Can AI truly "understand"?#

23.2Consciousness and subjective experience#

23.3The moral status of AIs#

23.4What AI teaches us about ourselves#

Key takeaways (chapter 23)

23.1Can AI truly "understand"?

23.2Consciousness and subjective experience

23.3The moral status of AIs

23.4What AI teaches us about ourselves