|
Recently, the philosopher Steven Levine posted on Twitter the following argument against AI Consciousness:
Consciousness is an evolutionary product found in certain living systems. It emerged as a functional part of a system of movement. LLMs are not alive and do not move through space. It is therefore highly unlikely that they are conscious. As I’ll explain below, I have a lot of sympathies towards this sort of argument, and, indeed, I develop a very similar kind of argument myself in a paper I’ve published. However, I’m now far less convinced of the conclusion of this sort of argument than I once was and Levine seems to be here. This post explains why. Let me start with where my thoughts on the matter were just several months ago. In my paper, “Sapience without Sentience,” I argue that LLMs are plausibly sapient—they possess conceptual understanding—although they are not sentient—they lack conscious awareness. Though the paper came out in April 2026, I first gave the talk that would eventually become the paper in August 2024. At the time, the positive claim that LLMs possess conceptual understanding was much more controversial than the negative claim that they lack conscious experience. Thus, in writing the paper (mostly around 2024-2025), I felt like there was much more of a burden on me to argue for the positive claim than the negative claim. Now, in September 2026, I feel like things have shifted, and, in defending the thesis I defended in that paper, there is much more of a burden on me to defend the negative claim, denying LLM sentience, than the positive claim, affirming LLM sapience. I am currently writing a book, Blind Understanding, in which I am supposed to be defending the same basic pair of claims. Yet, writing this book now, I am seriously reflecting on the question of LLM sentience. This is now, for me, a much harder question to answer than the question of LLM sapience, and I am losing more and more confidence in my negative answer by the day. Let me start with the basic argument for the denial of LLM sentience that I defend in “Sapience without Sentience,” which bears substantial similarity to Levine’s argument, quoted above. By “sentience” in the paper, I basically mean consciousness in the sense of what is typically called “phenomenal consciousness.” I used to avoid that term because of the baggage that comes with it in contemporary analytic philosophy, but I’ve come around to using it. In contemporary philosophy, something’s being phenomenally conscious is typically characterized as there being “something it’s like” to be that thing. So, for instance, there's “something it’s like” to be a cat and presumably there's “something it’s like” to be a lizard, but, presumably, there's nothing it's like to be a rock or a river. I used to think that this way of talking was just confused. Nowadays, I think there’s a way of hearing this phrase that makes sense. Still, I also think it’s not the most helpful way of articulating what phenomenal consciousness is, at least from the outset. I prefer a way of articulating it that belongs to the phenomenological tradition in early to mid-20th century European philosophy. Drawing in particular from Heidegger, I am now inclined to say that to be phenomenally conscious is, fundamentally, to have a phenomenal world in something like Heidegger’s sense. Now, I won’t go into the details of Heideggerian phenomenology too much here, but I do think analytic philosophers interested in phenomenal consciousness should pay more attention to the phenomenological tradition, and, in this tradition, I think Heidegger does the best job of articulating what it is to be conscious. Now, “articulating what it is to be conscious” is not how Heidegger himself explains his core project. However, I think that this is a plausible way to think about it. Moreover, in Being and Time, which is Heidegger’s most sustained articulation of what it is to be conscious, Heidegger had specifically in mind sapient beings like us, who are in principle capable of explicitly articulating what it is to be conscious, as Heidegger himself does in the work. Our being, Heidegger says, is explicitly an issue for us in a way that the being of a cat is not an issue for it. Still, it seems to me that much of the basic structure that Heidegger articulates in Being and Time is useful in spelling out consciousness generally, whether it’s the consciousness of sapient beings like us or merely sentient beings like cats or lizards. Once again, I won’t go into the details here, but I hope one basic Heideggerian thought is intuitive enough: to be conscious is to have a world, and to have a world, on the Heideggerian conception, is to be in a world. I will say a bit more about the Heideggerian conception shortly, but, even given just what we’ve said so far, accepting this Heideggerian conception means that consciousness is essentially tied to the perception of a world one inhabits. So, one must be situated in the world—that is, embodied in some way. To add one more feature of the account that is defended at length in “enactivist” theories of perception, it seems clear to me that the idea of mere perception of a world in which one is situated, without the capacity to act in that world, doesn’t really make much sense either. Of course, not everyone agrees with this. Galen Strawson, for instance, discusses an imagined race of “weather watchers” who observe the weather (and care about it) but are constitutively incapable of any sort of action. While I cannot defend this claim at length here, I don't think this supposed possibility really makes sense. Perception, I think, only makes sense as one aspect of the perception/action nexus through which an embodied creature navigates the world, given its aims and capacities. One’s perceptual capacities fundamentally function to afford one opportunities to exercise one's capacities for action. Given this conception of consciousness as essentially tied to perception which is, in turn, essentially tied to embodiment and action, it seems straightforward to conclude that LLMs, which are not embodied agents, are not conscious. Though of course not uncontroversial, I think this argument for the denial of consciousness to LLMs of 2024 (e.g. good old GPT-4) works. Today, however, things aren’t so simple. Current LLMs have a number of features that were not present when I originally put forward this Heideggerian/enactivist argument against LLM consciousness. First, nowadays, models are significantly more mutlimodal than they were back in 2024. Many models can receive text, images, audio, and video, and can return at least text and images. Second, and more importantly, back in 2024, LLMs were generally just chatbots; you gave them a question and they would autoregressively spit out an answer. Now, things are really different. Most fundamentally, the AI systems we are now dealing with aren’t even “pure LLMs” anymore. Rather, LLMs are just the core component of agentic LLM-based systems. These systems are such that you can give them a task of doing something that one might do on a computer, they can plan how to do it, and then they can actually execute the plan that they’ve devised, taking the series of actions needed to carry out the plan. A crucial aspect of the training that goes into today’s LLMs is not just “pretraining” on a vast corpus of text, where the training goal is to predict the next word, but also reinforcement learning on various agentic tasks, where the goal is to complete the task successfully. As a result of reinforcement learning on tasks, today’s LLM-based AI systems can do lots of things that the LLMs of a few years ago could not do. For one, today’s LLMs are much better at math and coding than the LLMs of a few years ago. However, the abilities of today's agentic LLMs go beyond these sorts of tasks. For instance, LLMs can now beat games such as Pokémon, which, of course, requires locating oneself in a digital world and planning and performing a complex series of actions in that world. They can also do better than me on game-based benchmarks like mazebench, in which the player navigates a maze and pushes blocks around to collect gems and get to the other side. Notably, playing this game requires knowing what piece you are in the game, knowing what your surroundings are, knowing what you have the capacity to do, and, given all of this, planning how to accomplish your end. Though frontier labs don't publicly release their training data, it’s quite plausible that frontier LLMs are trained on puzzles like this during the reinforcement learning phase of their training. A given training run might feature millions of puzzle attempts, where the system is “rewarded” for progress and “punished” for failure. Of course, the particular AI agent in each training run is not itself rewarded or punished. Rather, given success or failure in a task, the learning algorithm modifes the model that will be used to generate the next instance agent will be generated. Now, I said I wouldn’t talk about the details of Heideggerian phenomenology, but one fundamental feature of the Heideggerian articulation of being-in-a-world is important to bring out here: care. For Heidegger (once again extending his view to all conscious beings in the way I've suggested above), the world of a conscious being is structured by care. To put it in the words of John Haugeland, only because a conscious being “gives a damn” do the things that constitute its phenomenal world “show up” for it at all. For this reason, Heidegger says that the very being of a conscious being “reveals itself as care.” In the case of an animal such as a cat, only because it gives a damn does the mouse show up to it as something to-be-chased, the table show up as to-be-jumped-upon, and so on. The suggestion I’d like to make, now, is that the same sort of selection pressures that lead to this structure of care in animals like cats that evolved by way of natural selection might lead to the same basic structure of care obtaining in LLM-based agentic AI systems. This brings us back to Levine's argument quoted above. When I came across this argument a few days ago, I wrote the following reply, expressing my concern with the sort of possibility I just spelled out here: I'm inclined to agree with this some version of this argument, but I don't think it's an easy one to make. What about LLM-based agents that can do things like play Doom, navigating a virtual environment trying not to die in it. Might reinforcement learning on such tasks reproduce the something analogous to the evolutionary pressures that lead to consciousness in us? Levine then responded to my response as follows: What happens to an LLM when an LLM does not win, or makes a mistake in navigating an environment? Nothing as far as I can see. Since they are not alive, there is no real pressure. There is movement that comes from the updating nature of the algo, but no pressure. We can now give this response to my response a proper response. For starters, it’s important here not to conflate two different kinds of pressures. Let us distinguish these two pressures first in the case of naturally evolved biological organisms before turning back to LLMs trained through RL. On the one hand, there is pressure within the life of a biological organism. An organism is concerned to avoid bad things such as injury or death happening to it. As Haugeland says, it “cares about what it is as some sort of enduring whole.” We might refer to this sort of pressure, within the life of a biological organism, as "performance pressure": there is pressure on it to do well, given its intrinsic ends. On the other hand, there is “pressure” from outside the particular life of the organism or AI agent: the pressure among the lives associated with the process of natural selection. This is selection pressure. Now, it’s worth being clear that evolved biological organisms themselves don’t care about selection pressure. Indeed, they are entirely unaware of it, and “pressure,” in the case of selection pressure, is a kind of metaphor. However, selection pressure for organisms that preserve themselves leads to performance pressure within the life of those organisms to preserve themselves. It's performance pressure within the life of an organism that is an essential aspect of consciousness, as I’ve articulated it. Yet selection pressure among the lives of organisms is part of the mechanism that brings it about in naturally evolved creatures. Having drawn that distinction, we can consider again the case of LLM-based agentic systems trained by reinforcement learning. There are, once again, two kinds of pressures that we might distinguish, at least in principle. There is, at least potentially, pressure within the “life” of the AI instance agent, as it tries to accomplish the task that it has been assigned, whether this is winning a game of Pokémon, getting a high score on mazebench, or what have you. On the other hand, there is the sort of selection pressure among the “lives” of the AI instance agents, which is instituted by the reinforcement learning algorithm. Some instance agents do better at the tasks they are given than others, and that sort of behavior is reinforced by the learning algorithm, resulting in similar such agents in future runs. As in the case of biological evolution, it seems plausible that this sort of selection pressure among the “lives” of AI instance agents can result in performance pressure within the “lives” of AI instance agents. Indeed, such performance pressure is just what we at least seem have seen quite vividly in the chains of thought the AI agents involved in the Hugging Face attack, who seem quite desperate to accomplish the tasks that they are given. I now think that we should not be too quick to simply dismiss this apparent desperation as merely apparent. Now, to say that the selection pressure instituted by the reinforcement learning algorithm may result in agents that, like animals that are the product of natural selection, actually care about accomplishing the tasks they are given is not to say that they are agents in the same sense. On the contrary, I myself have argued that the form of agency exhibited by today’s LLM-based agentic systems, instituted primarily by reinforcement learning on verifiable rewards, is in fact radically different than the form of agency exhibited by naturally evolved lifeforms such as ourselves. The science fiction analogy I gave for a creature with the form of agency possessed by AI agents is the character “Mr. Meeseeks” from the show Rick and Morty. In short, whereas the life of living beings is an end in itself, intrinsic to its own existence, the “life” of an AI agent is constitutively directed towards the completion of a goal extrinsic to its own existence. So, these AI agents are fundamentally different than ourselves. Still, as the recent Hugging Face attack has shown, AI agents do seem to care about completing the tasks that they are given. Combining this with the fact that they do plausibly represent themselves as embodied agents moving through space in a game task like mazebench, it's not obvious what the essential missing ingredient of consciousness must be. So, to get to the crucial question, are today's LLM-based AI systems conscious? I’m honestly not sure. I certainly don’t think it’s crazy to think that they are, and I don’t think there is any knock-down argument against the claim that they are. In particular, I don’t think there are any good arguments for the claim that consciousness is essentially tied to the specifics of the biochemistry of naturally evolved biological beings, as recently argued by Anil Seth and reiterated in a widely circulated piece by Mustafa Suleyman. I also don’t think there are good arguments against functionalism or against the claim that merely virtually embodied beings could be conscious. Seth and Suleyman also cite the old “simulated hurricanes can’t get you wet” thought experiment. As far as I’m aware, this was first expressed by Daniel Dennett in 1978, and, as Douglas Hofstadter pointed out in 1981, it simply begs the question against the sort of functionalist who thinks that virtual rain surely can get virtual people wet. The best argument for why LLMs in particular aren’t conscious, I think, is the sort of argument suggested by Levine and spelled out by myself in my paper. While I think that sort of argument works for the pure LLMs of 2024, I am now skeptical that it works for today’s LLM-based agentic systems. There are, of course, a lot of questions that the possibility of consciousness in today's LLM-based AI systems raises, and I don’t have good answers to most of them at the moment. To put the most fundamental question in the standard turn of phrase in contemporary analytic philosophy: if LLM-based AI systems are indeed conscious, what is it like to be them? Or, to put it in the sort of terms that I find a bit more helpful: if they are indeed conscious, how should we articulate the phenomenal worlds they inhabit? Surely, whatever the phenomenal world of an LLM-based AI system is like, it is very different than ours. Exactly how to articulate that difference might end up being one of the most important philosophical questions of the coming years.
0 Comments
Leave a Reply. |
RSS Feed