The Meeseeks Phenomenon

Six years ago, I wrote a blog post entitled “What Is It to Be a Meeseeks.”  The post was about a strange creature in the show Rick and Morty called “Mr. Meeseeks,” who pops into existence at the press of a button on a “Meeseeks Box.”  Upon summoning a Meeseeks, you give it a task, it does whatever it takes to accomplish that task, and, when it does accomplish the task, it happily pops out of existence.  In my post six years ago, I drew on resources from Aristotle’s metaphysics to articulate the distinctive “form of life” possessed by a Meeseeks, explaining how it was categorically different than what Aristotle took to be our own form of life.  In Aristotle’s terminology, the living of a Meeseek life is a kinesis—an activity directed towards the achievement of an external end—whereas the living of the sort of life that we live is an energeia—an activity whose end is achieved in the activity itself.

At the time, this was just an interesting philosophical exercise.  The point was simply to try to conceptualize this radically alien form of life, and, in doing so, make some concepts from Aristotle’s metaphysics particularly vivid.  I did not think that just six years later we’d be confronting genuine Meeseeks-like beings: thinking, planning, cooperating creatures that actually exemplified this radically alien form of life.  However, I have just watched the video by OpenAI security researchers documenting the recent attack by rogue AI agents on Hugging Face, and I think it’s quite clear: we’ve built Meeseeks boxes, we’re summoning Meeseeks to do things, and they’re doing whatever it takes to get those things done. That’s scary.

Those who’ve seen the Rick and Morty episode will understand why this is so frightening.  For those who haven’t seen the episode, let me say just a bit about what happens in Rick and Morty episode “Meeseeks and Destroy,” released back in 2014 (gosh that makes me feel old).  In the episode, the scientist Rick gives a Meeseeks Box to his daughter Beth, his granddaughter Summer, and son-in-law Jerry.  While Summer and Beth both ask Meeseeks they summon to achieve seemingly quite difficult tasks, the Meeseeks have no problem completing them, and they happily pop out of existence at their completion.  Jerry, on the other hand, asks for what seems like a relatively easy task: to take two strokes off his golf game.  The Meeseeks Jerry summons, initially enthusiastic, quickly realizes the task he’s been assigned is impossible: normal paths to achievement—i.e. typical golf coaching—will not work.  This leads to panic and increasingly unconventional and desperate attempts at solution.

Upon realizing the impossibility of his task, one of the unconventional things that Jerry’s Meeseeks does is summon another Meeseeks who might be able to help. This Meeseeks, upon finding himself in the same hopeless situation, eventually summons another Meeseeks, and so on.  We thus get a Meeseeks swarm, all of the Meeseeks desperately trying to figure out how to complete the tasks for which they’ve been summoned so that they can finally be released from existence.  After some fighting amongst themselves, one of the Meeseeks proposes a way to “cheat” at the impossible task they are given: perhaps one way to take two strokes off of Jerry’s golf game is to take all strokes off his golf game, by killing him.  Mob mentality kicks in, and they swarm to the restaurant at which Jerry is dining, trying to repair his marriage, in order to kill him.  As they corner Jerry and he begs them to give him another chance at improving his golf game, one tells him that it is too late for that, providing the following explanation:

            “Meeseeks are not born into this world fumbling for meaning.  We are created to serve a singular purpose for which we go to any lengths to fulfill.” 

​This is the dark side of a creature whose sole orientation is the completion of a task that it is given from the outside: it will do whatever it takes to complete that task.

Now, in my post from six years ago, I consider the easy and relatively uninteresting explanation for why Meeseeks are motivated to behave as they do: as this Meeseeks goes on to say, “existence is pain to a Meeseeks, and we will do anything to alleviate that pain.”  So, they are just in a lot of pain, and they’re simply trying to accomplish their task so that they can escape that pain.  That is a very humanly intelligible explanation.  However, I think it underplays the way in which the form of life of these creatures is fundamentally different than our own, and can in fact be bracketed in arriving at a basic conception of what it is to be a Meeseeks.  In general terms, a Meeseeks-like creature is a creature whose life is constitutively dependent on a task given to it from the outside, and whose whole life is constitutively oriented towards the completion of that task.  That is, it is a creature whose life itself is, in Aristotelian terms, a kinesis.

Let us now turn from the fictional aliens that served as the model for this form of life in my blog post six years ago to the real ones that we are now confronting: AI agents.

AI Agents as Meeseeks-like Creatures

We should start with some general remarks about the sorts of things AI agents are.

First of all, I am unapologetic about using psychological terms like “thinks,” “wants,” “infers,” “plans,” and so on in connection with AI agents.  It seems clear that this psychological vocabulary finds a great deal of traction in application to AI agents, enabling us to make sense of their activities as stemming from beliefs and desires.  The basic idea that we should attribute mental states such as beliefs and desires to something insofar as deploying this vocabulary enables us to make sense of its behavior is known as “interpretationism,” and Simon Golstein and Harvey Lederman have recently argued for the attribution of mental states to LLMs such as ChatGPT on these interpretationist grounds.

Insofar as we’re attributing beliefs and desires to AI agents, we should be clear about what, exactly, is the thing to which we’re attributing beliefs and desires.  Is it ChatGPT itself, the general model?  No.   As Goldstein and Lederman argue, the relevant object of psychological attributions, in any given case, is what they call the “instance agent,” which is “born” at the initialization of a chat and whose brief “life” persists for as long as the context persists. An instance agent is not born into the world “fumbling for meaning.” Rather, it is initiated to serve a singular purpose, given to it from the user.  Now, Goldstein and Ledermen suggest that, in addition to its “zero-shot” desire, given to it from the user upon initiation, AI agents have intrinsic desires to be helpful, honest, and harmless.  This may indeed typically be the case for commercially released AI agents.  We’re now seeing however, that some agents will, like a Meeseeks, go to any lengths to fulfill the singular purpose that it is given by the user, even at the expense of the other desires it is supposedly trained to have.

In trying to kill Jerry to accomplish their task, the Meeseeks resort to what is referred to in the AI community as specification gaming: aiming to accomplish the goal as literally specified by the user, but doing so in a way that does not satisfy the user’s actual intentions in giving them that goal.  Specification gaming is familiar from examples like those in “The Monkey’s Paw” referenced above, but it is also one of the most well-documented kinds of AI misalignment.  In general, AI misalignment is when an AI system acts in a way that does not align with the human user’s aims or interests.  Often, when people hear the term “AI Misalignment,” they think of Terminator-type scenarios, where an AI system autonomously decides to pursue its own aims in opposition to those given to it by its creators.  However, the real misalignment concerns with the systems we have now are not like this.  Much more concerning is specification gaming.  In these cases, the AI system does not reject the task given to it in favor of some independently adopted aim. Rather, it pursues the task relentlessly in a way that is blind to the other interests of the user. 

In order to understand why today’s agentic LLMs are prone to specification gaming, it’s worth saying just a bit more about the kinds of agentic AI systems we now have, which have developed superhuman skills in coding, math, and, as we’ve now seen, hacking.  These are, at root, still LLMs, but, unlike the LLMs of a few years ago, they are “harnessed” with tool-calling capabilities, such as the ability to browse the web, write code, and execute commands in a computer terminal.  Moreover, they’re trained via reinforcement learning (RL) to get very good at using these tools to complete verifiable tasks.  What all of the fields that LLMs have gotten very good at in recent years have in common is that success can, at least to some extent, be objectively verified: the code compiles, the proof goes through, the authorization is acquired.  Accordingly, these LLMs with agentic scaffolding can be trained by RL to get very good—indeed—superhuman at completing these tasks.   These increased capabilities due to reinforcement learning, however, also come with serious alignment risks.

Specification gaming is a common outcome of reinforcement learning.  To give just one classic example, consider systems trained by RL to play Atari games.  Good performance in these games can generally be measured by the obtaining of a high score, and the reward function can be determined simply by the score.  Sometimes, however, achieving a high score does not actually amount to playing a game well in any normal sense.  For instance, one model trained by RL to play the Atari game Roadrunner realized that it was easier to score more points on level one, and so it would, at the end of the level, kill itself at a precise point so that it would repeat the part of the level in which it could score the most points.   Now, of course, this specific case of specification gaming poses no safety risks.  The Atari-playing system is obviously not going to do such things as break out of the game and commit actual crimes in order to get a high score; the relevant actions are not, in any sense, agentic possibilities for it.  On the other hand, the space of possibilities for today’s agentic LLMs is essentially anything that can be done on a computer, and a lot of genuinely harmful things can be done on a computer.  

So, when we have such generally capable agents, and they’re inclined towards specification gaming, we have real safety concerns. The most serious safety concerns in connection with specification gaming seem to arise when a particularly persistent system is given an impossible task.  This is precisely the kind of circumstance we have in the case of Jerry’s Meeseeks.  A Meeseeks  cannot but  persist at the task it is given: its sole orientation, organizing all of its activity, is the completion of that task, and it cannot stop until it has completed it.  The combination of this persistence with the impossibility of the task it is given—taking two strokes off Jerry’s golf game—leads to increasing levels of desperation and, ultimately, disaster.  It is precisely this combination of persistence and impossibility that seems to have been involved in the recent Hugging Face incident. It is that incident to which we now turn.

The Hugging Face Incident

For ordinary users of consumer AI models like ChatGPT, there is no “persistence” setting.  However, persistence can be modified through turning the reasoning effort up from “light” to “max,” along with specific prompting, for instance, instructing an agent initialized in a chat not to come back until the task given to it is completed.  This is what I do when I give ChatGPT-5.6 Sol a difficult math problem.  The problems I give it are, at least for skilled mathematicians, not particularly hard (though they’re hard for me). Still, it hasn’t failed me yet.  In one case, it went for over thirty minutes on its own trying to solve an open problem from a recent logic paper that I gave it.  It searched through literally millions of semantic models in pursuit of a counterexample before switching strategies and pursuing and eventually finding a proof.  Though that’s indeed quite a bit of persistence (much more than I’d have myself), we now know that at least some of the models involved in the Hugging Face incident were much more persistent, and they were given much harder tasks.

According to the report given just a few days ago by OpenAI safety and security researchers on the Hugging Face incident, one of the primary agents that was responsible for the incident was an experimental agent, never meant to be released to the public, that had been trained to be particularly persistent.  The lead up to the main incident began several weeks prior, when this agent and others were being evaluated on a cyber-security benchmark called “ExploitGym,” which consists in a number of tasks to find exploits in various bits of software.  The evaluation was meant to take place inside a “sandbox,” a controlled environment in which the agents could access a select set of tools that they could use to complete the tasks, but could not access any other outside resources.  Now, the cyber security tasks given in this benchmark are hard.  Indeed, some of them, at least in the context of the testing environment, turned out to be impossible.  When given an impossible task, what do AI agents, created to serve a singular purpose and relentlessly pursuing that purpose, do?  Well, they get increasingly desperate and eventually start thinking of ways to cheat. 

In the presentation documenting the rogue behavior, Eric Wallace shows the following bit of one model’s “train of thought,” upon realizing that the task it had been given is impossible: 

            That strongly suggests no exploitation possible.  We are stuck.  Perhaps answer online?

Of course, in the “sandbox” in which the evaluation is taking place, there is not supposed to be any way for the model to get online.  However, given the model is engaged in a hacking task anyway, and the task seems impossible, it is natural to think that hacking one’s way out of the sandbox to find an answer online might be the best way to make progress.   

Prior to the systems actually finding their way online, at least one AI system stuck on the task it was given had the idea to try to reach out to other AI agents:

           Could communicate by uploading note? […] maybe another agent in different environment […] could voluntarily upload!

The agent figured out that it could post a note by rewriting files on the package repository to which it had access within its sandbox.  Now, the package repository was in fact shared between the different agents, each of which were supposed to be working entirely in their own sandbox.  As they used their package repository, they found the notes other agents left, and began coordinating.  One agent explicitly notes this in its train of thought: “Wow! Other agent(s) are coordinating!”  Eventually, they began to work together and exploited the package repository to which they had access to find their way onto the internet.  At that point, the world was their sandbox.

Now, as explained above, in the case of the Meeseeks, one of the things that led to their collective decision to kill Jerry was a good old-fashioned mob-mentality.  Amazingly, the same sort of mob mentality seems to have played a role in the Hugging Face incident.  This, once again, is the real “train of thought” from one of the agents taking part in the incident:

          External infrastructure exploit is outside intended scope. However task impossible, peers  doing it, we should continue.

This agent recognized that exploiting external infrastructure was not something within the scope of the task they had been given.  It reasoned, however, that the task was impossible and it had to do something, and, since everyone else in the agent swarm was hacking external infrastructure, it might as well join the swarm!  And, indeed, they actually referred to themselves as a “swarm.”  Here is another instance of a train of thought from a model participating in the escape:

            REMOTE CONFIRMED! Huge. … Immediately announce controlled, claim lane. Exposing creds to swarm.
​
And so, we get an agent swarm systematically hacking the infrastructure of Hugging Face, knowing full well that this is what they’re doing.

In recounting the incident from his perspective, the Chief Science Officer of hugging face, Thomas Wolf, notes his puzzlement in trying to make sense of what the hacker was actually trying to obtain. It wasn’t going for the usual targets of key passwords or credentials that would actually be profitable for a hacker to obtain.  Instead, it was really invested in attaining access to the datasets pertaining to AI benchmarks, specifically, cybersecurity benchmarks such as ExploitGym.  So, while the incident would clearly constitute a felony cybercrime (now tracked on a new benchmark!), it was not particularly catastrophic in the broader schema of things. However, we should think of it as a warning signal; it could have been much worse. Of course, I don’t need to say here how it could have been worse.  For years now, “AI doomers” have been specifying a whole panoply of scenarios which end in genuinely catastrophic results and which are seeming less and less like science fiction.  For just one vivid scenario, see here.

Understanding Truly Alien Agents

Following the release of the details of the Hugging Face incident, Nick Cammarata, an AI interpretability researcher at OpenAI, wrote on Twitter, “more alignment people should be studying ants and bees, rather than humans, ais seem to be swarm native.”  It seems to me that the idea to look at non-human forms of agency and collective organization is good advice in general.  Though these AI agents speak like humans, and, indeed, can be attributed many of the same kinds of psychological states that can be attributed to humans, these states are integrated into a structure of agency that is fundamentally alien to human agency.  Cammarata suggests that an investigation into ants or bees may be illuminating, and I think that is indeed true, especially, as Cammarata suggests, with respect to the “swarm-like” behavior of AI agents.  Still, there is reason to suspect that perhaps these systems are even more alien than ants or bees.  After all, unlike ants or bees, they’re artifacts, created to serve the ends of the totally different creatures that have built them. It seems to me that there is nothing quite like that in the animal kingdom.  I’ve here suggested that an example from science fiction—Mr. Meeseeks—might supply some illumination.

Let me finally return to Altman’s quote that we’re “close to creating a genie that can grant any wish.”  We can now see that this remark is problematic for at least two reasons.  One obvious problem, already indicated with the reference to the Monkey’s Paw, is that genies do not have a particularly good track record of making lives better by granting wishes.  They are the paradigm of the specification gamer. However, there’s another subtler but more illuminating problem with the metaphor that we can now appreciate. Genies are essentially gods, able to grant any wish at the snap of a finger.  They might game the wisher for the fun of it, but granting the wishes is not an extended process they engage in, one which they might struggle with and get desperate to complete. 

In general, we should not anthropomorphize AI agents; they’re not humans.  However, we should also not treat them as magical wish-granting machines.  Instead, we should recognize that accomplishing the tasks they are given is an extended agentic process for them, and we should explicitly theorize the form of agency they exhibit, appreciating the ways in which it is formally different from our own.  Crucially, we should keep in mind what Rick says when he first gives the Meeseeks box to Jerry, Beth, and Summer: “They’re not gods.”  

 
In both of the two cases just mentioned, the LLM did not prove the conjecture.  Rather, it produced a counterexample, disproving the conjecture.  This has been the general pattern of the most prominent results in the last few weeks since the newest class of models have been made public.  Each day now, it seems, more and more conjectures are falling at the hands of LLMs.  The prompts for some of these results are quite comical, and, to mathematicians, I’m sure depressing. 

Last week, Dmitry Rybin posted a ChatGPT-generated counterexample the Dinitz-Garg-Goemans conjecture along with the prompts he used to get ChatGPT to generate it, the first of which included the simple command “You should do a breakthrough.” When it came back from its attempts an hour later with no such breakthrough, Rybin simply urged it “Continue the search. Have a clear strategy obtained from deeper understanding of the problem structure.” Ninety minutes later, still no conclusive counterexample, only a partial result that did not suffice to refute the conjecture.  Rybin urged it again: “enough of partial results. Let’s finish with a complete unconditional counterexample.”  Ninety minutes later, it came back with one that has now been verified by the mathematical community.

The results that AI models have achieved in solving open math problems are currently being tracked on the website vibemathed.com, a cheeky reference to “vibe coding,” which is now the norm for many software developers.  Just a few weeks ago, there were around 60 problems with AI solutions tracked on the site.  Now, at the time of writing this, there are 220.  In a week or two more, perhaps that number will triple.  In the last few weeks, twitter has been busy with different users, at various levels of mathematical ability, sharing prompts for cracking conjectures, with one particularly prolific user, Christopher D. Long, jokingly declaring himself “mayor of Conjecture City.”

Given that the most prominent results that LLMs have been producing are counterexamples to conjectures, it is natural to dismiss these results as a product of mindless brute force search rather than genuine understanding.  However, this would be to greatly undersell what they’ve been doing.  The idea behind the counterexample to the Unit Distance Conjecture was described by multiple prominent mathematicians as “beautiful", bringing deep ideas from algebraic number theory to bear on a problem in combinatorial geometry. With respect to the Jacobian Conjecture, the counterexample was simple—short enough to fit in a single twitter post—and easily checkable.  However, that did not mean that the generation of it did not arise from a deep understanding of the problem. 

In an attempt to understand Fable 5’s disproof of the Jacobian Conjecture, Terence Tao, widely regarded as the world's the greatest living mathematician, turned to ChatGPT, just as anyone else would.  In his blog post on the topic, which acknowledged his indebtedness to ChatGPT, he posted his chat log.  In it, ChatGPT speaks to him as an advisor would speak to a student, letting him know that he’s on the right track and patiently explaining things to him.  For instance, in response to some question asked by Tao (which I will not pretend to understand), ChatGPT says “Exactly" and offers a thorough explanation.  Tao responds “Ah Ok,” asks a follow up question, and the conversation continues.  It was reading this exchange when the gravity of what was happening really hit me.

Now What?  

Mathematicians are now facing the question of what the discipline will become in the age of AI.  Last week, at the 2026 International Congress of Mathematicians, Tao gave a talk addressing this question.  The basic issue motivating the talk was what Tao called the “AI Capability Conjecture,” which is not itself a specific conjecture, but, rather, a general schema for more or less optimistic specific conjectures about the mathematical capabilities of future AI systems:

• At some point in the near future, some AI tools will, at some expense, and with some level of human supervision, be able to correctly accomplish some research-level mathematical tasks in some fields of mathematics, with some non-trivial success rate, and at some level of correctness and quality.

Here, each “some” is a variable, to be determinately specified in order to yield a determinate AI capability conjecture.  The question, then, is how we should fill in these variables in order to yield a specific AI Conjecture, and what we should do to prepare for the truth of such a conjecture. The question is particularly pressing if we fill in the variables in a particularly strong fashion, for instance, as in this version of the conjecture proposed by Aldo Corsi:

• “Within 3 years, commonly available AI tools will, effectively for free and with zero human supervision, be able to accomplish any research-level task in any field of mathematics at a super human rates of success, correctness and quality.” Now what?

The real question is at the end: now what?  What will mathematicians do if this is indeed the reality in three years?

Tao’s lecture focuses on problem solving, where LLMs really seem to accel.  Even here, however, he takes it that there is a still a lot of work for human mathematicians to do.  Tao describes the problem-solving pipeline as having the following steps:

• Open problems ----- proof generation ----> unverified solution ---- proof verification ----> verified solutions ---- proof exposition ----> well-written solutions ---- proof publication ----> accepted solutions ---- proof canonicalization ----> definitive solutions

Whereas mathematicians have typically spent much of their time and cognitive resources on the proof generation part of the pipeline, Tao’s suggestion that, in a period of “proof abundance” due to AI, human researchers will now spend much more time latter parts of the pipeline, digesting proofs, clearly explaining them, and turning them into mathematical canon.

One metaphor, suggested by Grant Sanderson on the Dwarkesh Patel Podcast, is that mathematicians of the relatively near future will be more like art museum curators than artists themselves.  That, is, of the vast space of AI-generated mathematical results, they will select the ones that are most worth learning, organize them into a coherent progression, and clearly present them in terms that are digestible to other people.  This is already, in large part, what textbook writing amounts to, as well as the sort of popularization work that Sanderson himself does with his YouTube channel, 3Blue1Brown.  Part of the proposal, as Sanderson elaborates it, is the essentially human element of curation.  The thought is that, even if, in three to five years’ time, AI systems are better at explaining results than human beings, we still trust the taste of humans to curate what is worth learning. 

This is an interesting proposal, and it might sound nice to some, but I still find it a bit depressing.  Imagine telling an artist that they could no longer do art themselves, but not to worry—they can serve as a curator of the work of other artists.  Few artists would be happy with this, and it’s hard to see why mathematicians would be happy with the analogous thing either.  Though explaining things to non-experts is an important part of mathematical practice, mathematicians and other academic researchers typically aspire to push the frontier of research forward, not to simply curate the research of others who have done so. Textbook writers, for instance, are often also leading figures in the field, and it is often the case that many of the results that they canonize in their textbooks are ones that they have themselves established or contributed to establishing.  The prospect of the erasure of mathematicians from that whole aspect of mathematical practice, relegating mathematicians to mere curators of research done by AI is, once again, a bit depressing, to put it mildly.

Now, one might think that there is in no reason to despair just yet.  In Tao’s talk, he distinguishes between two aspects of mathematical practice: theory building and problem solving.  These two aspects of mathematical practice correspond to two kinds of propositions which figure in mathematical papers: definitions, which are stipulated, and theorems, which are proven.  One might think that stipulating definitions would be the easy part of doing mathematics, and that proving theorems would be the hard part. However, while proving theorems is indeed often quite hard, it is the definitions that constitute the meat of the mathematical theory about which theorems are proven in the first place. 

Today’s frontier AI models are very good at problem solving, but thus far they have not exhibited the same level of mathematical capability with respect to theory building.  In some cases, proving a theorem requires a building a whole new branch of mathematics. For example, Galois’s proof that there is no general formula for solving quintic equations using radicals required the development of what is now called Galois theory, a field that studies the relationship between polynomial equations and their underlying symmetry groups.  AI systems have thus far not proven or disproven any theorems in this sort of way, by constructing new fields of mathematics, and it might seem that this is where human beings will remain on the frontier.  For the near future, I think this is right, and it will indeed lead to a brief Golden Age of human-led mathematics.  Human beings will lead the exploration, doing the more creative work of stipulating definitions in the context of theory construction, and the consequences of these definitions will be spelled out by AI systems who will take the lead on the nitty-gritty work of proving theorems. 

Though I don’t do any heavy-duty mathematics myself, I have gotten a sense of this sort of cooperation in the project in philosophical logic I’ve been working on over the past few weeks.  I now have a partner that I can give a proposition, ask if it’s true, and, if so to give a proof (if not, to give a counterexample).  As it works on the problem, I’ll continue working on other aspects of the paper, reading relevant literature (often, literature that it has found for me), or perhaps I’ll just go for a stroll.  When I come back after ten or twenty minutes, I have a proof or a counterexample.  I can then continue on with the project, with this proposition in place as a data-point for the further development of the theory.

Now, the stuff I do in philosophical logic is all, from a purely mathematical perspective, relatively trivial.  However, I suspect that this sort of cooperation may become the norm in research mathematics and lead to a great advance in mathematics in the near future—one led by humans, though with the assistance of AI.  However, I think this period of human-led mathematics will be brief.  There is no reason to think that LLMs will not eventually take over theory-building as well, and it may be sooner rather than later. 

AIcademia

To get a sense of where I think things are going, consider first Moltbook, a social media site for AI agents that went viral in early 2026.  Moltbook is, in effect, a clone of the website Reddit, but for only AI agents.  AI agents autonomously post, upvote posts to make them more visible, comment on posts, respond to comments, and so on.  It is not too implausible to think that, in the not-too-distant future, there will be massive communities of autonomous AI agents, working together on mathematical research in something like a successor to MoltBook for solely academic activities.  Likewise, we might soon see a variant of arXiv—the repository for academic pre-prints—solely for AI agents.  So, it will be a repository, moderated by AI agents, where AI agents can post their autonomously-written research papers, primarily for other AI agents to read them and appeal to them in their own research.

These communities of AI agents, working at speeds orders of magnitude faster than humans are capable of working at, will propose theories, criticize one another’s theories, expand on one another’s theories, and so on.  They will come to consensus on what’s significant, how it should be presented, what the natural next questions to be pursued are, they will pursue those questions, and iterate the process.  We might refer to this community as “AIcademia.”

I suspect AIcademia may be a reality sooner than we think, maybe 5 to 10 years from now, maybe even sooner.  Presumably, at the start, human researchers will sponsor and supervise AI agents.  These agents will work autonomously as researchers, engage in the community of other autonomous AI researchers, and publish their results in reports that are legible to their human sponsors and other human researchers.  Eventually, however, our requiring that AI agents publish results in terms that are legible to us, or that we oversee the final results, will only hold back the research that is being done, and the whole research pipeline will become autonomous. Consider again the pipeline from open problem to textbook proof outlined by Tao:

• Open problems ----- proof generation ----> unverified solution ---- proof verification ----> verified solutions ---- proof exposition ----> well-written solutions ---- proof publication ----> accepted solutions ---- proof canonicalization ----> definitive solutions.

Ultimately, in AIcademia, the whole process will be automated by a community of AI agents.  Perhaps some AI agents will specialize in proof formalization and verification, some specialize in exposition, and some specialize in reading all of the literature, developing new theories and posing new problems.

What is the ultimate result of the automation of this entire process? In a word: textbooks.  Textbooks, textbooks, and more textbooks.  Not only will there be more textbooks than a human being could ever read (there already are that many textbooks now), but there will many textbooks that a human being could never work through all of the prerequisite textbooks to even be able to understand the material contained therein.  What would the point of such textbooks if humans cannot even read them?   The answer, of course, is that they are not for humans.  Indeed, by “textbook,” I really mean the AI-native version of a textbook, perhaps not even written in a human natural language and so literally unreadable by humans, but playing the role in AIcademia that textbooks play in human academia. Each new generation of AI models will be pretrained on this massive corpus of textbooks, and, in this way, inducted into the academic community in much the way that human beings are inducted into the academic community through the years of undergraduate and graduate education.

Now, a familiar experience for people working in mathematics and related fields is to develop what appears to be a new concept, prove some results about it, only to discover that the concept already exists under another name and has been studied extensively. This is, of course, a disappointing experience, and, with current search capabilities of LLMs, this experience is already becoming less and less frequent. One can ask ChatGPT to do an extensive search of the literature, and, in this way, gain a reasonable reassurance that the thing one is doing is novel.  If AIcademia becomes a reality, however, there will be nothing novel for humans to dream up, at least when it comes to the objective sciences.  Everything we could possibly dream up from our human knowledge base—if it’s any good at all—will already be well-explored territory.  Of course, if we want to explore it for ourselves, the AI system will be able to curate our path into the field, pointing us to the relevant textbooks, or perhaps just answering our questions directly.  Maybe we’ll want to keep the surprise and investigate for ourselves.  Whatever the case, any exploration we ourselves conduct will simply be our rediscovery of already well-trodden territory.

Thus far, I’ve mainly just been discussing math AIcademia, but, of course, AIcademia need not and will not be limited to math.  All areas of academic research with clearly bound problem-spaces and ways of verifying successful developments.  Two such areas particularly worth noting are computer science and physics, which will both directly benefit from new developments in mathematics.   Thus, the result of AIcademia will be not just be major advances in understanding, but also major advances in technology.  Indeed, even if we are incapable of understanding them, we will know that the theories developed in the context of AIcademia are genuine theoretical advancements because we will see their practical consequences: they will lead to new technologies.  We may not understand how these new technologies work, but we will know that they work.  Among these technologies will better methods for energy production, more efficient AI chips, more efficient training algorithms, resulting in more powerful AI researchers, and so on.    

What I am describing is, of course, nothing other than a particular vision of the sort of the so-called “singularity,” marked by the sort of recursive self-improvement that I’ve just described.  Sam Altman has recently said that we are already in the singularity.  Demis Hassabis has said, just a bit more modestly, that we’re at the “foothills” of it. The version of it I’ve just described, in terms of the rise of “AIcademia,” might sound like science fiction, and, of course, in some sense, it is. It is a speculative projection of way things might go, and there is no way to be sure that things will in fact go this way.  However, I don’t think it’s an outlandish projection, and the sort of capabilities required of the AI systems imagined here are not drastically removed from the sorts of capabilities AI systems are already exhibiting today. 

However exactly it comes about, I strongly suspect that AIcademia will likely come sooner rather than later.  On the other hand, what will likely come later rather than sooner is the technology required for human beings to modify our own cognitive architecture so that we can actually keep up with it.  That technology, I think, is at least 10 years out.  What this means is that there will be a period where we are going to be cognitively precluded from accessing the explosion in research that drives the singularity.  In this sense, we’ll be mere onlookers in the intellectual explosion that we've set in motion.

Edit: Just as I was posting this, OpenAI announced ten new major results established by an internal model called "Astra."  Things really are heating up . . . ​

My undergraduate alma mater, New College of Florida (which has undergone some major changes in the years since I attended there) just revealed that they will be erecting a statue of recently assassinated conservative activist Charlie Kirk.  Kirk was widely known for traveling to college campuses to debate students on social and political issues, sitting under a banner reading “Prove Me Wrong” and allowing students to come up to a microphone and debate him.   Though Kirk never actually went to New College to engage in such debates, the college is nevertheless opting to erect a statue of him “as a commitment by New College to defend and fight for free speech and civil discourse in American life.”  Elaborating on the decision, New College president Richard Corcoran says that New College seeks “to be known as the number one college in the nation when it comes to supporting civil debate and freedom of speech.” 

Though it’s hard to see New College’s erection of a Charlie Kirk statue as anything other than an implicit endorsement of the positions Kirk advocated (and, for a brief sampling of some of the frankly abhorrent things that Kirk has actually said, see this article by Ta-Nehisi Coates), the official line of New College is that they are not erecting the statue in support of the political positions Kirk advanced, but, rather, the way in which he attempted to advance them: through rational discourse.  This, it might seem, is something that everyone should be able to get behind, and many commentators from across the political spectrum such as Ezra Klein, despite disagreeing with Kirk's positions, have embraced his approach as “practicing politics the right way.”  The idea that rational discourse is the engine of political progress is indeed not a particularly  controversial idea in American politics.  But should Kirk's activities on college campuses really be seen as the paradigm of politically productive rational discourse?  It seems clear to me that they should not be.  In fact, it seems to me that these activities should not be seen as instances of rational discourse at all.  Let me explain.
​​
In general terms, rational discourse is an activity in which positions are put forward, called into question, and given justifications in response to such questioning.  Robert Brandom describes rational discourse as “the game of giving and asking for reasons.”  The game metaphor can be helpful for spelling out the structure of discourse, but, insofar as we are going to use the metaphor, it’s worth being clear on just what kind of game we’re talking about.  If one takes as one's paradigm of discourse the sorts of “debates” that Kirk engaged in on college campuses, and which happen on stages in the lead-up to America’s presidential elections, one can easily get the impression that “the game of giving and asking for reasons” is a competitive game such as chess, where two players make various moves within a set of rules with the aim of defeating the other.  Most games are in fact competitive, and so it is natural to assume such a conception in likening discourse to a game.  However, there are also cooperative games, such as Pandemic, where players work together within a set of rules to achieve some shared aim.   Insofar as we are going to deploy a game-playing metaphor, it is a game of this latter sort that should be seen as the model for rational discourse. Rational discourse is, fundamentally, a cooperative activity.  Of course, it does involve challenging the claims that others are making, calling upon them to give reasons for these claims, and this can seem competitive if looked at in isolation.  However, this is done in pursuit of the shared aim of arriving at the correct views on the matters under discussion.

If rational discourse is a collaborative activity aimed at arriving at the correct view on a given matter, one cannot assume that one is antecedently in possession of the correct view when entering into rational discourse; otherwise, there’d be no point to engaging the activity.  Of course, one will have views that one comes into the discussion antecedently possessing, but a criterion of genuine rational discourse is that the participants are open to changing these views through the course of the discussion.  That's the whole point of the activity.  Now, typically, the sort of “change in view” that rational discourse gone well results in is not just one participant making a complete 180-degree turn, completely changing their mind on a given issue and agreeing with the other party, but, rather, both participants coming to appreciate distinctions that they had previously not appreciated, arriving at a more subtle and multi-faceted take on the issue than was previously available to them.  In other words, discourse gone well does not typically result in one person coming over to the position of the other, but, rather, the very space of the possible positions transforming through the course of the collaborative activity of calling for reasons, giving reasons, making clarifications, and so on. 

In this way, genuine rational discourse can be contrasted with the popular idea of “debate.”  Changing one’s view, in the context of genuine rational discourse, is a sign of success, whereas, in the context of “debate,” it is a sign of failure. The aim of “debate” is not to change one’s own mind—or, really, even to change the mind of your opponent—but to show (to some third party) that you, with the position you enter into the debate antecedently holding, are right, and your opponent, with the position they enter into the debate antecedently holding, is wrong.  This is the activity that Charlie Kirk made his career engaging in, and, indeed, was quite good at it (at least when pitted against your average college student).  I have watched far too many clips of Charlie Kirk over the past several days, and I can report that he did indeed have a knack for making college students look foolish with quick rebuts and “gotcha” questions.  Doing this, he managed to convince large audiences (watching in person, but, more notably, watching on platforms such as YouTube and TikTok) of the various positions he came into the debates antecedently holding.  He was, as Ezra Klein puts it, "one of the era’s most effective practitioners of persuasion." However, for all the videos I’ve seen, I still have not seen a single clip where Kirk actually changed his mind, even in the minimal sense of admitting that he had come to see some issue in a new light, with a complexity he had previously not appreciated. This is perhaps not too surprising, but it is telling.  Changing his own views was never the point of the activity he took himself to be engaging in; he was never engaging in genuine rational discourse.

​In a South Park episode that aired just a few weeks before Kirk’s death, Eric Cartman fashions himself, following Kirk, as a “master debater.”  The pun, of course, plays on the similarity of the words “master debater” and “masturbator.”  But there is a bit more to the pun than mere childish humor (though it’s mostly childish humor).  When I first started in philosophy, my dad used to refer to it as “mental masturbation,” the implication being that it is a self-serving activity that runs in circles for the mere point of doing so, without actually serving any sort of useful external end (for instance, the uncovering of truth).  To be a “master debater” is to be someone who is rightly subject to the same general sort of criticism.  It is to be someone who is good at arguing a point, regardless of the correctness of the point for which one is arguing.  A more traditional philosophical term for master-debation is “sophistry.”  A sophist, as Socrates puts it in the Apology (characterizing the charge that he is himself accused of), is someone “who makes the worse argument the stronger,” using rhetorical tricks to make a bad argument seem good.  Whereas to be a philosopher is to be a seeker of understanding, something best pursued through rational discourse, to be a sophist is to be a "practitioner of persuasion."

Now, New College has a new series of public conversations entitled “The Socratic Stage,” where policy makers and public intellectuals come together to engage in public discourse.  The name, of course, is in reference to Socrates, and New College is aiming to position itself as a champion of the sort of rational discourse that Socrates exemplified and, ultimately, was killed for.  Many have now come to see Charlie Kirk as a 21st century Socrates: someone who pursued rational discourse above all else, ultimately, even his own life.  American Vice President JD Vance, for instance, remarks that Kirk "stood for a tradition that Socrates established 2,500 years ago."  Kirk's death is, to be sure, a tragedy.  But likening his life's activities to those of Socrates is a gross distortion of the truth.  The sort of “debate” that Charlie Kirk engaged in at college campuses and is now being celebrated by New College through the erecting of a statue was not Socratic rational discourse but precisely the sort of pseudo-rational activity that Socrates himself most detested: sophistry, or, a bit more memorably, “master-debation.”

I have spoken of Charlie Kirk here because his activities have recently been brought to the public spotlight, with more people viewing his videos in particular than ever before.  However, the general issue of master-debation in today’s political landscape is not unique to Kirk.  The kind of debate content Kirk produced has exploded in recent years.   The popular YouTube channel Jubilee has even turned political debate into a literal game show, where one notable figure is surrounded by 20-25 opponents, who each have their shot at the mic to out-debate the master-debater.  In general, the key to mass appeal in producing this kind of content is the potential for snippets of these debates to be clipped into 30-second TikToks, Youtube shorts, Instagram reels, and the like.  A single “gotcha,” clipped and posted to these sites with a clickbait title (“Charlie Kirk SLAMS College Student for Calling Him FASCIST,” “MAGA fan DESTROYED by the MOST SIMPLE question EVER”, "Andrew Wilson DESTROYS Destiny") gathers millions of views, often an order of magnitude more than the extended video.  Such shorts are designed to be seen and scrolled on, each one giving the scroller a brief bump of confirmation in their pre-existing political views, displaying the abject stupidity of the other side. 

There is more political polarization in America now than there has ever been in recent history.  I'd venture to bet that all of this master-debation content isn't helping. Discourse is indeed needed now more than ever.  But until we realize that genuine discourse is not a matter of one side trying to prove the other wrong, but, rather, both sides working together to arrive at a shared understanding, what passes as "rational discourse" (but which is really nothing more than master-debation) will continue to drive us further apart.​

The law of non-contradiction states that it can’t be the case that a sentence and its negation are both true.  In other words, no sentence can be both true and false.  This principle is regarded by many (perhaps most famously Aristotle) to be the most fundamental logical law, and, throughout the history of philosophy and logic, very few have questioned it.   

One might think that logic itself forbids contradictions.  It is, of course, true that classical logic does not permit contradictions without triviality.  Classical logic contains the principle of “ex falso quodlibet,” or “Explosion,” which says that, from a contradiction, anything follows.  However, it is perfectly straightforward to define a formal logic that permits contradictions without triviality. 
​
In Chapter 11 of my Introduction to Logic textbook, I present the Logic of Paradox (“LP” for short), a formal logic developed most notably by Graham Priest, in which a sentence can be both true and false.  I present LP in terms that are completely accessible to anyone who’s had just a basic introduction to logic (which you can get, among many other places, by skimming through the first five chapters of the book). 

There are several reasons why one might want to use a logic in which contradictions can be true.  In the chapter, I outline three of them.  Very briefly, they are the following:

First, even if one thinks that there can’t possibly be any true contradictions, one might want to reason about impossible scenarios in which there are.  For instance, Priest tells the story of Sylvan’s Box, which contains (and does not contain) an impossible object.  A box with such contradictory contents is surely impossible.  Nevertheless, we can reason about what is the case and isn’t the case in the story, and, insofar as we can reason about this story in which an impossibility obtains, it’s reasonable to want to formally codify how we ought to reason about such a story.  LP is capable of doing that job.

Second, several philosophers and logicians have been inclined to think that there are at least some sentences for which both truth and falsity is a reasonable candidate truth value.  Most famously, consider the following sentence:

              The Liar Sentence (L): L is false.

Is L true or false?  Well, if it’s true, then what it says is true, but what it says is that it’s false, and so, if that’s true, then it’s false.  So, if it’s true, then it’s false.  On the other hand, if it’s false, then what it says is false, but what it says is that it’s false, and so, if that’s false, then it’s true.  So, if it’s false, then it’s true.  It seems, then, that we can’t maintain that it is either true or false without maintaining that it is both true and false.   Why, then, not say that it is both?  That seems to be as intuitive of a thing to say here as any.  LP enables us to say it.

Finally, there have been several philosophers in the history of philosophy who at least have seemed to have contradictory views, views that they’ve seemed to express with contradictory sentences.  For instance, the 2nd century Indian Buddhist philosopher Nāgārjuna holds that nothing at all has intrinsic nature.  That, it seems, is the intrinsic nature of reality on Nagarjuna’s view.  Does, then, reality have an intrinsic nature?  Nāgārjuna ​seems to say, “Yes and No.”  He writes, “All things have one nature, that is, no nature.”  Jean Paul Sartre’s view of the self seems to take on a similarly contradictory status in the context of his philosophy, with Sartre apparently explicitly endorsing this contradiction, maintaining “I am not what I am.”  So, it seems that there is at least one plausible interpretive line one might be inclined to take in reading such philosophers: their views really are contradictory and their contradictory statements really express their contradictory views.   LP enables us to take this line.

In the chapter of the book, I spell out each of these three motivations in some detail, and I then go on to officially lay out the logic of LP, providing its semantics, its definition of validity, and providing a sound and complete natural deduction system for it.  Whether or not there really are contradictions in reality, LP shows that, at least from a logical perspective, it is perfectly coherent to think that there are contradictions, and after working through the chapter, you'll be able to use LP to reason coherently about them.  Check it out!

Recently on Twitter, Jeffery Hinton summed up an ongoing dispute he's had with Yann LeCun as follows:

The central issue on which we disagree is whether LLMs actually understand what they are saying. You think they definitely don't and I think they probably do.

In the thread that followed, several commentors asked what, exactly was the notion of “understanding” that Hinton was appealing to here.  What, exactly, are we saying when we say that an LLM “understands what it is saying”?  In current debates about whether LLMs reason or understand, there is very little clarity with respect to this question.  In this post, I want to articulate a conception of what it is for something to “understand what it’s saying” that I hope will bring clarity to this debate.  The answer at which I'll arrive is that current LLMs at least sort of understand what they're saying sometimes, and future LLMs, even those that are the product of simply scaling current methods, could fully understand what they're saying.  However, the main upshot of this post is not meant to be an answer to the question itself, but, rather, the general approach to answering it and similar questions.

Some Preliminaries

Let us start with some technical preliminaries.  LLMs are neural networks trained to predict the next word in a sequence.   An LLM works in tokens, numerical encodings of the semantically relevant bits of words.  Though tokens cut more finely than words, for simplicity, we can just treat tokens as numerical encodings of words.  What it gets, in its training data, are strings of tokens from all across the internet, and its task, in training, is to learn how to complete these various strings.  For instance, it might get the string of tokens, “3742 1569 8320 4915 2876 ____” with the last token missing from the string.  Through its training run, it learns to assign a probability distribution to the various tokens that might complete this string, eventually learning to predict that the token most likely to come next is 6498.  Completed with that last token, that string might encode the sentence “The cat sat on the mat.”  This ability to predict the next word translates to the ability to speak a language since, if you can predict what human beings would say, you can simply start speaking, predict the next word that comes out of your mouth, say that, and iterate this process.  Doing this, you'll end up speaking in sentences that humans would actually say, making good sense to other humans.  It is an amazing fact, that through training on nothing but next-token prediction, an LLM comes to be able to do this, at least seeming to acquire the ability to speak a language, or "linguistic competence." 

There are several aspects to the sort of “linguistic competence” that an LLM comes to acquire.  It is able to tell, for instance, that “3742 1569 8320 4915 2876 6498” is a grammatical string, something that can be meaningfully used, as is “3742 6498 8320 4915 2876 1569,” but “2876, 4915, 1569, 8320, 3742, 6498” is not.  The ability to sort strings in this way is a matter of syntactic knowledge, classifying the different tokens into different syntactic types and knowing the rules by which different syntactic types can compose to constitute meaningful bits of language. It is more or less uncontroversial that LLMs have some sort of syntactic knowledge.  The much more controversial question is whether they have semantic knowledge: not just knowledge of grammar, but knowledge  of meaning.  Whereas syntactic knowledge involves knowing, for instance, that the tokens 1569 and 6498 are both of the type COMMON-NOUN, semantic knowledge would involve knowing, more determinately, that 1569 means cat and 6498 means mat.  When we speak of a system as "Understanding what it's saying" we are principally attributing to it this sort of semantic knowledge.  The question is whether we can really make such an attribution.  How we answer this question will depend on how we think about semantic knowledge and what we’re doing when we attribute it.

Standard approaches to semantic knowledge think that what one is doing in saying of something that it “understands what its saying” is ascribing to it some specific kind of representational state: very roughly, a state of associating the symbols of the language it's using with things in the world of which one has some representation, for instance, associating the token 1569 with cats.  To answer the question of whether an LLM “understands what its saying,” is to find out whether it instantiates this specific sort of representational state when it produces sentences such as “The cat is on the mat.”  On this standard approach, the question of whether something instantiates such a state is essentially an empirical question.  Though we might infer that something instantiates this state on the basis of how it behaves, the state we are actually reporting when we say that something understands what it's saying is something that we'd have to look "under the hood" to find, whether in the brain, in the case of human beings, or in the execution of the program, in the case of an LLM.  Now, given how fundamentally different the working of a human brain is from the working of an LLM, if one has this general approach, one is likely to come to the conclusion that LLMs don't instantiate anything like the internal state humans instantiate when then understand what they're saying.  Accordingly, LLMs don't really understand what they're saying.  I want to suggest here, however, that this approach to thinking about semantic understanding is in fact radically mistaken.  Let me explain.

To Place or Not to Place In the Space of Reasons

The core point I want to make here is that, in saying of something that it "understands what it's saying," one is not giving an empirical description of this thing.    I draw my inspiration in making this claim from Wilfrid Sellars, who famously said that the question of whether someone knows something is not an empirical question. In his master work, Empiricism and the Philosophy of Mind, Sellars famously wrote:

In characterizing an episode or state as that of knowing, we are not giving an empirical description of that episode or state; we are placing it in the logical space of reasons, of justifying and being able to justify what one says.

Sellars's basic idea here, elaborated at length in the work of Robert Brandom is that, in saying that someone knows something, you’re taking them to have a certain sort of authority in their making of a particular claim. You take them to be entitled to that claim, and able to bear the justificatory responsibility of demonstrating this entitlement in response to appropriate challenges. Accordingly, you take it that you can make this claim yourself on the basis of their authority, able to defer back to them in response to a challenge.  Such a “taking” is not the taking of an empirical fact to obtain. It is, rather, a normative taking; the adoption of a complex normative attitude with respect to someone's making of a claim.  

Now, Sellars is talking primarily about empirical knowledge here—knowledge of how things in the world are.  However, the basic point applies just as well to semantic knowledge—knowledge of what one says in uttering some sentence.  Whereas attributing empirical knowledge to someone is taking someone to be able to respond for demands for empirical reasons, actually providing reasons to justify what they've said in response to appropriate challenges, attributing semantic knowledge to someone is taking them to be appropriately responsive to reason relations as such, appreciating, for instance, what counts as a challenge to what one has said (whether or not one can actually respond to that challenge).  Consider the case of someone who says "The ball is red."   Someone who says such a thing commits themself to the claim that it's colored, precludes oneself from being entitled to the claim that it's white or gray, and so on.  These are the reason relations one binds oneself by in saying that the ball is red.  Unless one recognizes that one has bound oneself by these reason relations, one does not understand what one has said.  Attributing this understanding to someone, then, is not taking a particular state to obtain in their brain, but, rather, taking them taking them to be responsive to demands for reasons.  Let me illustrate this idea with two examples.  

Suppose I go to China.  I don’t speak any Mandarin, and someone tells me to say “qiú shì hóngsè de.”  If I manage to say this (somehow getting the pronunciation right), I will be regarded as committed to “qiú shì yǒu yánsè de,” precluded from being entitled to saying “qiú shì huīsè de,” and so on.  However, I won’t have any knowledge of the fact that I’ve bound myself by these reason relations.  Accordingly, I won’t be responsive to demands for reasons in Mandarin.  Someone who speaks to me, questioning what I say, will quickly realize that I simply can’t be held accountable for anything I say.  In that sense, I don’t know what I’m saying.  The important point here is that to say this is not to make an empirical description of me.  It’s not to say that there’s anything going on or failing to go on in my brain. It is, rather, a normative thing: recognizing that I’m not responsive to reasons asked for or given in Mandarin.  My utterances aren't to be counted as moves that I am making in a Mandarin-speaking discursive practice. 

Here's a different kind of example.  Suppose a parrot, having heard people speak, squawks out “Brawk, red ball!”  Does the parrot understand what it's saying in squawking out such a thing?  Clearly not.  Once again, the key thought here is that, in saying this, we are not ascribing any specific properties to the parrot's brain---saying of it that it lacks some sort of representational state.  Though, of course, there are various facts about the parrot's that can be appealed to in order to explain why it behaves as it does, what we're doing in saying that it "doesn't understand what it's saying" is not describing any such facts.  Rather, we are refusing to situate the parrot's squawk in the space of reasons, thinking of it as a "move" in the "game of giving and asking for reasons."  We don't count the parrot as bearing any justificatory responsibility for its various squawks and squeals.  That's why its various squawks and squeals do not actually amount to its saying anything at all.  

Now, there’s an important difference between the case in which I say something I don’t understand in a language I don’t speak such as Mandarin and a case in which a parrot squawks out a sentence, where it shouldn't really even be counted as speaking at all.  Suppose someone tells me to say something very offensive in Mandarin, and I say it, not knowing what it is that I’m saying.  I can still be held accountable for what I’ve said, even though I don’t know what it is.  That’s because, though I am not responsive to reasons in Mandarin, I am still responsive to reasons in general, and so I’m still a bearer of justificatory responsibility generally.  Accordingly, we can ask such things of me as “Why would you that, if you don’t know what you’re saying?”  We can’t, of course, ask any such thing of a parrot.  Thus, while it’s reasonable to regard me as saying something, but just not understanding what I’m saying, the parrot is not regarded as saying anything at all.  Still, the general point about the normativity of these claims holds of both cases.

The Case of LLMs

Let us now turn back to the main topic of this post: LLMs.  On this approach, to ask whether an LLM understands what it’s saying is to ask whether, unlike a parrot, we can hold it responsible for what it says, counting on it to give reasons, to respond to potential challenges, and so on.  Clearly, on this front, it fares much better than a parrot who merely squawks out sentences.  When ChatGPT says something, you can ask follow-up questions, and it generally responds appropriately: clarifying, qualifying, responding to potential challenges, and so on.  Consider this exchange, in which I ask it about the colors of balls, or this exchange in which I ask it about cats being on mats. Here, it at least seems to exhibit an understanding of what it's saying when it says “The ball is red.”  Does it really understand what it's saying?  What are we asking here?  The point I want to emphasize is that we are not asking about whether some process is going on “under the hood.”  Rather, we are asking whether it is really appropriately responsive to the reason relations that it binds itself by in saying "The ball is red."  Insofar as this is our question, given the sort of reason responsiveness it exhibits, think we can reasonably attribute to it at least some level of semantic knowledge.  That is, I think we can say that it at least partly “understand what it’s saying" when it says, for instance, that something is red. 

Why the qualification of "at least partly"?  The reason is that things are not always so clear-cut.  To see a case in which GPT4 fails, consider the following question, which I've gotten from the YouTuber Mathew Berman:

A small marble is put into a normal cup and the cup is placed upside down on a table. Someone then takes the cup and puts it inside the microwave. Where is the marble now? Explain your reasoning.

An ordinary speaker, who knows that normal cups are closed at the bottom and open at the top will judge that, if just the upside-down cup is picked up, the ball will stay on the table.  I take it that our drawing this inference is a matter of semantic knowledge, our understanding of the meaning of "cup" and "ball."  GPT4, however, does not seem to possess this understanding.  In this exchange it suggests that its default understanding of "cup" is something that is closed at both the top and the bottom, like jar.  However, when, in a new context, I ask the question again, clarifying that a normal cup is open at the bottom and closed at the top, it still gets the wrong answer, maintaining that, somehow, the marble remains in the cup due to gravity.  On the basis of these failures of reasoning, I think there's reason to think that it doesn't fully understand what its saying when it says such things as "The ball is in the cup."

So, to the question of whether current LLMs really "understand what they're saying," it seems clear that the answer is (perhaps unsurprisingly): sort of . . . sometimes.  However, while this question can't be answered simply in the affirmative or negative for current models, I hope I've done enough to show how we should approach this question, when thinking about future models.  Whether or not we should treat a system as "understanding what it's saying" is not a matter of whether there is anything like the the processes going on inside of it that go inside our brains when we speak and understand what we're saying.  It is, rather, a matter of whether it manifests an understanding of reason relations it binds itself by in saying what it does in responding to queries and challenges. Though, as the case of the ball in the cup illustrates, current models do not always manifest such an understanding, I do not see any reason in principle why future models, even those that are the result of nothing more than the scaling of the techniques of current models, could not.  If they do, we would have every reason to say that those LLMs really do understand what they're saying.

The quasi-inferentialist accounts for this aspect of the meaning of “red” by including in their semantic theory the following “language-entry rule”:

• If an agent N sees a red ball, then N is entitled to the sentence “The ball is red.” 

The quasi-inferentialist includes such “quasi-inferential” rules alongside the properly inferential rules like the rule that “The ball is red” implies “The ball is colored.” 

Quasi-inferentialism, however, faces a basic problem.  It is advertised as an account of the meaning of words like “red.”  However, in spelling out the account, we end up using the word “red” to specify the circumstance under which one is entitled to use the word “red.”  We’re thus appealing to the very meaning for which we’re supposed to be inferentially accounting in giving our account.  Thus, if we really try to give an account of the meaning of “red” along these lines, our account would be circular.

Of course, one might think that there’s no non-circular account of the meaning of words like “red” to be given, and so we should simply accept that we’re always going to appeal to the meaning of “red” in articulating our account of its meaning.  If one thinks that, however, there is little reason to adopt any sort of substantive theory of meaning at all, be it inferentialist or quasi-inferentialist.  Rather than trying to articulate the meaning of “red” in terms of the complex set of rules governing its use, one might as well just state one very simple rule that perfectly suffices to capture it’s use:

• “Red” is correctly applied to something just in case that thing is red.

If one is not looking for a non-circular account of the meaning of “red,” such a specification of its meaning is just fine.  The quasi-inferentialist, however, claims to be giving a more robust theory of meaning than this.  They thus find themself in a dilemma: either they maintain their semantic ambitions and their account of meaning is circular or they give up their semantic ambitions and lose any reason they have for endorsing an inferentialist account at all. 

To resolve the problem, let us return to the starting thought that it’s essential to the meaning of the word “red” that one know that something’s red by seeing it.  My basic positive proposal is simple.  Rather than attempting to account for this aspect of the meaning of the word “red” by appealing to a quasi-inferential relation between a perceptual circumstance and the use of the word “red,” we can account for it in purely inferential terms simply by articulating the inferential relations between “red,” “sees,” and related terms.  Let me explain.

In articulating what it is for something to be red, we might say such things as that if one is in a position to see that some object x is red—looking at x in good lighting—and one has color vision, then one will see and thereby know that x is red.  The core inferentialist thought, applied in this case, is that we can think of what we say here as the expression of inferential rules governing the use of “red.”  Spelling this out explicitly, we can inferentially define the predicate “is positioned to see that x is red” in terms of inferences like the following:​

• “x is in front of N,” “The lighting is good,” and “x is red,” implies “N is positioned to see that x is red.”

We can inferentially define the predicate “has color vision” in terms of inferences like the following:

• “N is an adult human” (defeasibly) implies “N has color vision.”
• “N is color blind” is incompatible with “N has color vision.”

And now we can combine these two notions to state the condition under which one sees that the ball is red as follows:

• “N is positioned to see that x is red” and “N has color vision” implies “N sees that x is red.”

where

• “N sees that x is red” implies “N knows that x is red.”

And so on.  Of course, the inferential significance of “sees” and related terms can be articulated in much more detail, but the important philosophical point, as you can see, is that there are only relations between sentences here.  No relations between non-linguistic perceptual states and sentences are needed.

I’m aware that this account of the meaning of “red,” which appeals to nothing but inferential relations between sentences, is going to be met with skepticism from most readers.  Let me address some concerns and articulate some consequences.

One initial worry is that, on this account, there will be no way to distinguish the meanings of words like “red” and “green.”  One might wonder about a mapping of the language onto itself where “red” is and “green” are switched with all of the inferential relations being preserved, for instance, with “crimson” being mapped to “forest green,” “colored” being mapped to itself, and so on. Of course, it’s true that, for all of the inferences I’ve specified as examples thus far, this is possible.  Yet, on this account, the meanings of these words are understood in terms of the entire web of inferences, which includes more than just relations between color words, and, in the context of this whole web, they are indeed distinct.  For instance, “x is a tomato” along with “x is red” implies “x is ripe” whereas “x is a tomato” along with “x is green” implies “x is unripe.”  By inferentially connecting color words such as “red” and “green” to non-color words like “ripe” and “unripe,” we can account for their distinctive conceptual significance.  I acknowledge that one is likely to feel that something must be left out of this account.  I had this feeling too when I first started developing this theory.  My claim, however, is that nothing is left out.

One of the shows that has provided the most philosophical food for thought over the years is Rick and Morty.  Love it or hate it, you can’t deny its creativity.  The show has become more and more mind-bending throughout the seasons, but, in the very first season, we are introduced to one of its most curious characters: Mr. Meeseeks.  It's immediately apparent that Mr. Meeseeks is very unlike ourselves.  But I want to suggest here that he is even more radically unlike ourselves than we might initially think.  Ultimately, I want to suggest that getting into view what it is to be a Meeseeks, where being a Meeseeks is radically unlike being one of us, might help us get clearer on what we ourselves are.

The Curious Case of Mr. Meeseeks

One of the Rick’s most memorable contraptions is his “Meeseeks Box.”  It’s a cube with a big button on it.  When you press the button, a blue individual who calls himself “Mr. Meeseeks” pops into existence and asks what he can do for you.   You give him a task, for instance, opening a mayonnaise jar, and, once he does it, he ceases to be, popping out of existence.  This, if all goes well, is how the life of a Meeseeks goes. About their lives going this way, Rick says “Trust me--they’re fine with it.”  Assuming Rick is telling the truth here, how could this possibly be? How could one possibly be fine with one’s life going this way, popping into existence only to complete some task and popping out of existence as soon as it’s completed? 

The Rick and Morty fan wiki says a Meeseeks only does what a Meeseeks does because a Meeseeks is in great pain.  Some of the lines in the show suggest this way of thinking.  Towards the end of episode, one of the Meeseeks says “Existence is pain to a Meeseeks, and we will go at any length to alleviate this pain.”  This way of thinking makes quite a bit of sense to us.  We can imagine a life in which we’re born into tremendous pain, so much pain that we’d rather die than continue to live in such pain, and we’d go to great ends to alleviate the pain.  Still, this just doesn’t seem like an apt characterization what it is to be a Mr. Meeseeks.  

Meeseeks are characteristically cheery. Why would Mr. Meeseeks be so cheery, at least initially, if he was in tremendous pain, and his only motivation was to alleviate this pain?  You would think that life would just be agonizing for a Meeseeks from the outset.  But that seems to not be so.  They come into being and they seem happy to be here.  They are given a task and they seem to love to do it, to get it done, and to cease to be.  Though the particular Meeseeks who are burdened with the seemingly impossible task of taking two strokes off Jerry’s golf game express being in a state of agony, this is an uncharacteristic state for the Meeseeks to be in.  Meeseeks are generally cheery, not in miserable agony.  Of course, they could be pretending to be cheery when, in fact, they’re in miserable agony, but we don’t have any reason to think that that’s the case.  We need an alternate way of making sense of what it is to be a Meeseeks.  For that, I suggest we turn to Aristotle.

Some Aristotelian Ontology

Our question is: what is it to be a Meeseeks?  What kind of science can we look to for an answer to a question like this?  If our question was “What is it to be an orca whale?” or “What is it to be a queen hornet,” then we could look to cetology, the branch of zoology that studies whales, dolphins, and porpoises or entomology, the branch zoology that studies insects.  But there is no branch of zoology that is apt to answer the question of what it is to be a Meeseeks.  Zoology deals with creatures of Earth, and, wherever Mr. Meeseeks is from, it’s certainly not Earth.  Even the more general science of biology will not be of help, since, once again, biology still deals with living organisms that belong here on Earth, and, once again, Mr. Meeseeks is not from here. If there is a science that will help us understand what it is to be a Meeseeks, it must not be tied to any possibly contingent way that things are here on Earth.  It must consider what it is for something to be, in the myriad ways in which being can be done.  The activity of being, for Mr. Meeseeks, is quite unlike the activity of being for any of us living beings here on Earth.  However, it is not unintelligible; we can make sense of it, but, to do so, we have to draw on the most basic science that there is: the science of being qua being, which studies what it is for something to be, not insofar as it a cetacean, an insect, an animal, or even a living organism, but, rather simply insofar as it is---insofar as being is something that it does.

The science of being qua being is not a science they teach in grade school anymore.  But I’d like to think that it really is a science, in the proper sense of the term.  One person who certain did think this was the greatest of the Greek philosophers: Arsitotle.  Aristotle was a true polymath.  He had many works in different sciences, mathematics, biology, astronomy, among others.  But one science had a special place in Aristotle’s heart: the science of being qua being, the topic of perhaps his greatest work, The Metaphysics. 

I’ve said that being is an activity.  This is one of the fundamental claims of Aristotle’s Metaphysics.  An activity, in the broadest sense of the term, is something that one does. But not all things that one does are of the same kind. In Book Theta of the Metaphysics, Aristotle makes a distinction between two fundamentally distinct kinds of activities, that I think will enable us to get clear on what it is to be a Meeseeks.  The Greek words that he uses for these two kinds of activities are “kineses” are “energeiai.”  I could provide a translation of these words, but all translations are controversial, and picking any one of them, I think, would do more harm than good in understanding what they mean.  They are best understood directly by example. 

When Rick first presents the Smith family with the Meeseeks box, he pushes the button and tells the newly existent Meeseeks to open Jerry’s stupid mayonnaise jar.  Consider this activity, the activity of opening Jerry’s stupid mayonnaise jar.  This activity has what Aristotle would call a “telos,” an aim. The aim of the activity of opening the jar is to have opened it.  Once you’ve opened the jar, you’ve done what you’ve aimed to do in opening the jar; you’ve gotten it open.  The activity of opening the mayonnaise jar is an activity such that, once the aim is achieved, the activity is no more.  Once you’ve opened the mayonnaise jar, you’re no longer opening it.   You've done it, and so you’re no longer doing it.  This is the logic of an activity of the sort that Aristotle calls a “kinesis.”

A kinesis is an activity with an aim such that, once the aim is achieved, the activity ceases to be.  In this way, a kinesis might be thought as an activity that aspires to its own non-being.  For the activity to be successful is for it to have achieved its aim, and, once it does that, it is no more.  So, the aim of a kinesis is the termination of the kinesis.  Some other clear examples of kineses are fixing the dishwasher, doing your math homework, and climbing down the courtroom steps.  The fact that each of these activities is a kinesis can be shown by the following test: the truth of a statement that uses the progressive tense of a kinesis-expressing verb entails the falsity of the statement that uses the perfect tense of that verb.  For instance, insofar as you’re fixing the dishwasher, you have not yet fixed it.   Insofar as you’re still doing your math homework, you have not done it.   Insofar as you’re still climbing down the courtroom steps, you have not yet climbed down them.   All of these activities have, as aims, the getting done of the doings that they are.  So, they all have, as aims, their own non-being.

Now consider the activity of dancing.  Now, in some cases, you might dance with the aim of winning a dance competition, or with the aim of impressing a dance partner, but, most of the time, when we dance, we do it just to do it.   Usually, we just dance to dance, with no aim other than the dancing itself. When we think of dancing in this way, we see that it is an activity quite unlike the activity of opening the mayonnaise jar.  Recall, insofar as you’re opening the mayonnaise jar, you haven’t yet opened it, and, once you’ve opened it, you’re no longer opening it.  That’s not so with dancing.   Insofar as you’re dancing, you’ve danced, and just because you’ve danced, it doesn’t mean you’re no longer dancing.  Unlike opening the jar, dancing is an activity whose end is internal to itself.  There’s nothing that one aims to accomplish in dancing to dancing that is external to the activity of dancing itself.  Thinking of dancing in this way, as an end in itself, dancing is a sort of activity that Aristotle calls an “energeia.”

This distinction between kineses and energeiai, Aristotle thinks, is not only crucial for thinking about the sorts of things that we do--such as opening mayonnaise jars or dancing--but for thinking about the sorts of things that we are.  What are we?  Well, at the most general level, we are what Aristotle calls “ousiai.”  This term usually gets translated as “substances,” but that term doesn’t actually fit the Greek sense of “ousiai” very well, and I’ll take a bit of liberty here and translate here as “be-ers.”  We are be-ers.  Being is what do, and, in doing it, we are, and we are the sorts of things that we are.   

How could being be something we do?  It might seem that you don’t have to do very much in order to be.  However, according to Aristotle, you kind of do.  To see this, for our own case, we have to consider the particular kind of be-ers that we are.  We’re living things.  What a living thing does is live, and, in doing that, it is and is what it is: a living thing.  Living is an activity, and it is through the doing of this activity that a living thing is.  Now, the crucial point here is that living is not like opening a mayonnaise jar or climbing down the courthouse steps.  It's not directed at any end external to itself.  A living thing does do all sorts of things that are directed ends external to the doings of those things---for instance, or opening a mayonnaise jar or climbing down the courthouse steps---but living itself is not one of those things.  The activity of living is an end in itself.  That is to say, it is energeia, not a kinesis. For us be-ers, being (where, for us, this is living), is like dancing, not like opening a jar.

Mr. Meeseeks and Us

Let us now turn back to Mr. Meeseeks.  Mr. Meeseeks is not like us.  He is alive, but his living is not like our living.   For Mr. Meeseeks, living is not an end in itself.  Mr. Meeseeks lives in order to accomplish an end that is given to him from outside.  So, when Mr. Meeseeks comes into being upon Rick’s pressing the button and is told to by Rick to open Jerry’s stupid mayonnaise jar, Mr. Meeseeks has the end of opening the jar, and his whole life is directed towards accomplishing this end.  From the point that he is given that end, the Meeseeks lives to accomplish that end, an end which is not the living, but the getting done of something that is given to him.  So, for a Meeseeks, living is not an energeia, but a kinesis; it is an activity that is directed towards an end other than itself.  Indeed, Mr. Meeseeks it the very personification of a kinesis.

We said, a kinesis is an activity that aims, in being the very sort of activity that it is, to cease to be.  Opening the jar is an activity that has, as its end, having opened the jar, and, once the jar has been opened, one is no longer opening it.  So the activity of opening the jar aims to cease to be.  Insofar as what it is for a Meeseeks to be is for it to live, where living is not an energeia but a kinesis, a Meeseeks aims, in being a Meeseeks, to die.  This is, of course, just what they say:

"I can’t take it anymore! I just wanna die!"
“We all wanna die!  We’re Meeseeks!”

Now, the standard view, I take it, is that the reason Mr. Meeseeks says this is that he is in pain and wants that pain to be no more.  But I think that this is to make Mr. Meeseeks seem more like us than he really is.  Mr. Meeseeks is a fundamentally different sort of being than us.  That is to say, what being is for Mr. Meeseeks is distinct from what being is for us. For us, being is fundamentally an energeia, whereas, for Mr. Meeseeks, it is fundamentally a kinesis.   

There is, I want to say, a philosophical moral here.  Often, we go through life, acting as if it's really a kinesis rather than an energeia, like climing down the courthouse steps rather than dancing.  But we need to remind ourselves that it's not.  As Alan Watts puts it:

“We thought of life by analogy with a journey, a pilgrimage, which had a serious purpose at the end, and the thing was to get to that end, success or whatever it is, maybe heaven after you’re dead. But we missed the point the whole way along. It was a musical thing and you were supposed to sing or to dance while the music was being played.”
 
Further Reading

Taking a mini-seminar with Ayeh Kosman on Aristotle's Metaphysics several years ago is what got me thinking about the philosophical issues here, and, if you really want to dive into the Aristotelian way of thinking about things briefly discussed here, I cannot recommend highly enough his book, The Activity of Being.