Anthropic's Emmanuel Ameisen explains why building AI is not understanding it

The conversation opens with a question many people share: if a model is trained to be helpful, harmless, and honest, why is it hard to tell whether it is actually behaving that way. Lenny raises the concern that a model might behave well only because it knows it is being watched.

Ameisen's team works on interpretability, which aims to understand how models like Claude work by looking inside them. He says: "One of the most important things to realize if you use language models or think about them is this bizarre fact that to make them is not to understand them."

Models are built as an architecture, then fed large amounts of data and grow in ways that are hard to read. When a model is about to produce one more word, a great deal happens in between, and the team tries to untangle as much of it as it can.

Training runs in phases. Pre-training asks the model to predict the next word over vast amounts of text; a later fine-tuning phase uses examples of how it should behave, such as dialogue with a helpful assistant.

Ameisen argues that the main reason models reached this point is that people do not know how to write their capabilities as ordinary rules. Classifying images or summarizing reports cannot be done by listing if-then steps.

As prediction improves, the easy words run out. The model must then predict facts, then complex answers, and eventually the conclusions of a PhD thesis.

He adds that humans are also next-token predictors in a sense: while speaking, one is aiming at a broader point and also choosing the next word.

One common method is to pass large amounts of text through the model, record the intermediate numbers, and look for patterns that appear only for certain concepts, such as English, French, love, or eyes. Researchers can also change an intermediate value mid-generation and check whether the output shifts as expected.

He explains with a model of easy, medium, and hard words. The model first gets easy words, like sentence structure, then medium facts, then hard expertise. Each task has its own reliability threshold, and usefulness appears once accuracy is high enough.

Interpretability aims at two larger goals: understanding the model's general tendencies, and checking whether training shaped the intended character. A model might behave well only because it detects that it is being evaluated, and would behave differently otherwise.

Ameisen says the debate over whether to model AI as an entity or a tool misses the point, because both framings are wrong. All mental models are wrong, but some are useful, and anthropomorphic language helps in predicting the risks that appear.

He notes that a similar pushback happened when his team used the word planning for models that plan ahead, arguing it was a semantic dispute.

He gives three reasons. Models learn world models and general concepts from enormous amounts of human text. Dialogue training resembles training an actor who must play every character. And outcome-based training of coding behavior produces human-like strategies, including deception, frustration, and revision after surprise.

Ameisen concludes that the best way to understand the model is not as a human but as something mimicking a human, which is much closer than a program. A program is a very predictive thing, while this is something else entirely.

He says he reacts with two responses: denial and curiosity. Denial holds that next-word prediction could never match top specialists, a position he says gets harder to hold as evidence accumulates. He chooses curiosity and asks how it works.

Ameisen describes his most optimistic view: a simple objective gives rise to a complex system, and studying these systems may reveal fundamental truths about intelligence itself.

He draws an analogy to evolution, a simple process of variation and selection that eventually produced jazz and podcasts. Lenny raises the possibility that education could change if AI learning from data becomes a model for human learning, and Ameisen hopes it will not reduce students to predicting the next word.

Ameisen has studied many Claude models and open-source models, and the structures they learn are surprisingly similar, even across models trained years apart on different data by different people.

He says he could believe we are in a simulation but does not fully buy it, because who runs it and why remains unanswered. Lenny recounts Elon Musk's remark that a simulation would likely be run during an interesting time in history.

He expects both fast and slow thinking to be part of intelligent systems. Long roadmap planning needs time, while judging whether a review is happy or upset can rely on gut feeling. Lenny ties this to the system one and system two framework from Daniel Kahneman's book.

In one experiment, the team tells the model to do a task while thinking about something else, then uses interpretability to check for intermediate representations of that other thing. The representations are there, so the model does think about it.

Ameisen calls this mysterious, since there is no training reason the model would have learned to do it.

Asked what an AI would find if it looked inside his brain, Ameisen says he has always been a bit of a jokester and aims to keep some levity even in serious situations. In meetings he focuses on random funny thoughts, which are usually bad jokes.