Lance Martin on Making Claude Perform Better: Prompt Audits, Caching, Evals and Avoiding Overfitting

Martin said cost and quality are usually treated as a trade-off: better performance requires spending more, and cutting cost means sacrificing performance. On any given evaluation, models form a curve he called the Pareto frontier.

He said that many applications are not actually on this frontier but sit inside it, and attributed this to minor misconfigurations, which he said he would show how to fix.

He said he has seen this pattern repeatedly across customers, where a few simple changes improved performance and reduced cost.

He outlined five themes and said the rest of the session would be a question-and-answer format. The first theme was auditing prompts.

Martin said that in Claude Code you can run checkup prompt audit, and that Claude API offers prompt audit as well. Both point to the same underlying skill, which audits anything in the Claude Code working directory, such as skills, tools and prompts.

In his demo, the migration alone produced a small cost saving and a slight performance gain. Running the prompt audit on top of that gave a further improvement in both performance and cost, he said.

He attributed this to the command removing verification rituals and boosters that were causing Opus 5.5 to spend excessive tokens, and to resolving contradictory prompts that had been confusing frontier models.

He recommended running the command, saying Anthropic uses it extensively internally and many customers have liked it. It can be run through the API or in Claude Code configuration.

Martin pointed to the Claude API build eval command. He said it is not tied to where your data lives, so you can point it at whatever tracing solution you have. It is also a skill in Claude Code.

He said that given a problem, your interests and raw traces, the skill shows you the data, shows the evaluation, and sanity-checks the grader and criteria with you, guiding you through building an eval set. He mentioned a related blog post.

His criteria for a good eval were that it has headroom at the frontier, low variance, and scores rise as expected with more thinking and model size.

Hill climbing is a process in which Claude looks at an evaluation and improves on it autonomously, without a person in the loop. Martin said frontier Claude models are very effective at this. Once you build a good eval, Claude can climb on it.

He warned, however, that many things can go wrong, in particular overfitting to the evaluation itself. For example, if one benchmark task needs OCR and the harness adds an OCR tool to solve it, that tool may be absent in production and the gain disappears, because the process has fit the eval distribution.

He said another common case is prompts tuned to distinctive phrasing specific to the eval examples. Overfitting, he said, is the major challenge when hill climbing.

He also described a failure case: an evaluation score looks very good after hill climbing but performance in production does not improve at all, wasting a lot of tokens and money. The hill-climb command splits the eval into test and train sets to reduce this, but he said it is not perfect, so a person must audit the results and understand what is happening.

Martin said hill climbing can also be applied to cost reduction. If you tell it to reduce cost, it looks for a configuration that is at least at parity in performance and definitely reduces cost. It searches effort, model choice and prompts.

In a customer-support benchmark example, he started with Opus 4.8 high and a baseline prompt. The hill climber searched model and effort combinations, keeping those that kept cutting cost over rounds without degrading performance. Moving to Opus 5 low gave a large performance gain and a minor cost reduction.

The hill climber then tried 5 low autonomously, which slightly degraded performance but cut cost substantially. It then tuned the prompt, which recovered the performance. The result was a roughly 90% cost reduction with a notable performance improvement, he said.

He said this also works when an eval is saturated and performance no longer matters, since you can tell the hill climber to hold performance fixed and reduce cost.

Martin said Claude Managed Agents never gives Claude direct access to secrets. Secrets are held in vaults, which are stored in the tools box at the top of the diagram.

Because of the vault, Claude can reference API keys or secrets but cannot actually see them. He said that for an agent working for days on end, giving it direct access to sensitive secrets would be a little spooky. He added that as agents operate for hours, days and perhaps weeks, reliability and security become more important.

Asked how automatable eval design is, Martin said Claude helps significantly. He said people are spending much more time on eval design, mainly specifying tasks that Claude can effectively hill climb on.

He recounted feedback from Hamel, who said the skill should walk the user through the data first. Martin agreed and said he would modify the skill. In a customer-support example, Claude was good at scanning traces to flag failure cases, acting like an online evaluator scanning offline logs.

He said an intermediate stage matters: Claude should first show the raw data in a viewer before building the eval, and he has a pull request up to add that. Humans still promote examples into the eval set.

Martin said Claude is good at catching repetitive work and suggesting it become a skill. He said he does this all the time and that you can simply tell Claude to turn a workflow into a skill.

For evals, he said the usual setup is an existing skill with boilerplate text, and he asks Claude to improve it so it performs better on the eval. He gave an example of a Claude API skill distilled from the docs, which had many gotchas and so scored low on the eval at first. The starting point let Claude modify it.

Martin explained that by distribution he meant making sure the cases sampled for an eval represent what you actually care about in production.

He cautioned against adversarial sampling, where you select only the cases one model class fails. Models are jagged, so Sonnet 5 may fail examples that Opus 5 solves, and an eval shaped to one model's valleys may not apply after you upgrade models.

He therefore recommended promoting cases that are hard or interesting from first principles, from your own intuition and from production data, rather than sampling only from one model's failures.

He also said that grading on an arbitrary 0-to-10 scale can be unclear, and that pairwise comparison can be more effective, for example asking which of two webpages is better rather than grading each from zero to five.

Martin said there is no guarantee the hill climber will converge, and that he has often seen it get stuck. To address this, they built a fallback: if it stays stuck for several rounds, it clusters the remaining failure cases to find out why they fail.

He said that often the last 20% are faulty grader examples or ambiguous instructions, so the climber is stuck not because Claude cannot do better but because the eval is poorly structured. In his own run, the second stage got stuck, the cases were clustered, several ambiguous ones were found and rewritten.

He also said Claude is very good at building UIs and visualization surfaces, so for a bespoke problem you can cheaply build an app to look at the data.

Martin explained that effort tells Claude how hard to think: low effort means think less, and scaling it up means think harder. He warned that starting at max effort is extremely token-hungry, so it is often wise to start with a smaller, smarter model at low effort and go from there if needed.

He gave the intuition that Sonnet 5 at high effort spends many tokens thinking, while a smarter model at low effort spends fewer tokens. So a more fundamentally intelligent model at low effort can be more token-efficient and therefore cheaper than a cheaper model at high effort.

On budgets, he said that when you set one for hill climbing, the skill manages the budget per round. He cautioned that a poorly specified eval, for example one whose final 20% is unsolvable, burns many tokens on problems that cannot be solved.

Martin described an eval for generative UIs in which a pairwise judge compares configuration A and B to decide which is better, the results are rolled up across many trials and scored with an Elo score.

He said there are many other examples he could not recall, but people use LLM-as-judge all the time, for retrieval tasks, writing tasks, categorizing traces and judging traces as good or bad.

He said the motivation for hill climbing draws on auto research: models are very good at self-correction, so given a well-structured problem with a clear verifiable objective, they can improve against it.

In the example, Martin jointly optimized cost and performance. He said you can tell Claude the multiple objectives and it will measure and take them into account. Cost, performance and speed were a reasonable trio to measure together.

He said as you add objectives, the hill climbing becomes harder because Claude is balancing many objectives at once. It is still doable, and one option is to change one thing at a time and proceed serially.

He noted that prompt optimization is the simplest thing to try and often very effective. He added that it is not precise which anti-patterns frontier models no longer need, though it may be that something like Opus 4.5 still needs some guidance.

Martin said the process is philosophically similar to training, since you give the model a reward or objective and it modifies the system to improve it. Practices like test and train splits used in training also apply here.

The difference is that these are run-time modifications to prompts or skills after a model is deployed, so you can apply training-style principles in your own domain.

Asked whether Claude knows it is being evaluated, he said this is a concern that has been published on in training. For this setting, though, he said the bigger concern is overfitting: Claude may know it is being evaluated, but will find ways to hill climb the objective based on the eval distribution, such as prompt tricks that do not apply in production.

Martin said that when building agents you typically need multi-turn evals. He said they are harder because of the infrastructure lift, the failures that occur, and the longer time to get a signal.

He said the build eval command can look at traces and help promote particular traces into the eval set. For grading there are many philosophies: evaluators can look at the whole trajectory or only the final agent outcome.

On context engineering, he said a key to reliability is decoupling brain and hands: the context object is independent of the harness, so the harness can query that context. He said they employ that idea in their own work.

Asked how to modify evals as models keep improving, Martin said that as models get better, performance on an eval should go up as in the diagram. If it does not, he would go back and question how the eval is structured.

He said models saturate evals very quickly, so new and harder evals must keep being developed, and that it is actually hard to build evals that challenge today's best models.

His key point was that if you have evals built some time ago, you should see their scores rising, and if not, you should question the eval itself.