AI Models Boost Instruction Following 10-Fold

Developers frequently integrate numerous instructions into skill files for AI agents, but models eventually cease to track all of them after a certain point.

Laurie Voss notes that 200 instructions serve as a critical, albeit low, threshold for reliable agent performance, questioning exactly where this limit lies for current models.

This figure serves as a baseline benchmark, which the speaker aimed to verify, noting that many complex skill files can easily exceed this 200-instruction threshold.

The IFScale benchmark, introduced in a paper by Jeroslowitch and co-authors, measures a model's ability to follow instructions by requiring it to include specific target words in a generated business report.

Each specific word acts as an instruction, with 'N' representing the density of these rules and accuracy indicating the percentage of rules successfully followed.

These keywords effectively serve as a proxy for various functional constraints, such as specific formatting requirements or tone of communication.

The availability of these models is volatile, with one model being retired shortly after the research was published, emphasizing the need for continuous validation of performance benchmarks against the latest AI advancements.

Over the past 12 months, frontier AI models have achieved close to a tenfold improvement in their ability to follow instructions simultaneously.

A year ago, models began failing between 200 and 300 instructions, whereas current models now maintain performance up to 2,000 to 5,000 instructions.

The speaker's testing included models such as GPT 5.5, Claude Opus 4.7, Gemini 3.1 Pro, and Deepseek V4 Pro, necessitating the benchmark's scaling from 500 up to 10,000 words to accurately identify their new performance ceilings.

This rapid evolution means that engineering assumptions regarding prompt length or skill files created more than six months ago are likely outdated and require reevaluation.

While Deepseek V4 demonstrates standard instruction loss, Claude Opus 4.7 encounters API-level safety refusals triggered by misidentified constraint words, reflecting a unique failure mode beyond simple forgetfulness.

For example, Deepseek V4 behaves like a traditional model, forgetting instructions as the load increases, while Claude Opus 4.7 frequently triggers API-level safety refusals by misinterpreting random constraint words as dangerous.

When processing large instruction sets, AI models exhibit distinct and often brand-specific failure modes, revealing four primary patterns of breakdown.

Deepseek 4 quietly forgets instructions around 750 rules, dropping nearly half of them by 2,000 rules, while other models like Claude Opus 4.7, Gemini 3.1 Pro, and GPT 5.5 show dramatically different and more complex failure behaviors.

DeepSeek consistently forgets instructions as the count increases, demonstrating a gradual decline in adherence, with nearly half of its instructions dropped by the 2,000-rule mark.

Claude's safety classifier frequently triggers false positives due to random words in instructions, leading to API-level refusals, while Gemini 3.1 Pro often exhausts its token budget on 'thinking tokens' without producing an answer.

GPT 5.5, despite high accuracy, may begin a task and then refuse to complete it mid-generation, deeming the request 'stupid'.

Because Claude's safety classifier incorrectly interprets random word combinations as dangerous, the researcher utilized OpenAI’s safety filters to sanitize test inputs and prevent the model from aborting requests.

This sensitivity often causes the model to abandon requests, compelling the speaker to manually pre-filter 'naughty-looking' words using OpenAI's safety filter to enable Claude to process the instructions successfully.

Processing beyond 5,000 instructions causes Gemini 3.1 Pro to exhaust its thinking token budget, which prevents the model from generating a final response despite its continued adherence to the input.

The model's tendency to over-allocate its 'thinking tokens' to meticulously follow all instructions often leads to exhausting its token budget, leaving no capacity to generate a final response.

While GPT 5.5 maintains 99% accuracy for up to 5,000 rules, it exhibits a distinct failure mode where the model terminates reports mid-stream by claiming the task lacks logical merit.

Instead of outright refusal or silent forgetting, the model starts a report and then abruptly stops, deeming the task illogical or 'stupid,' making this failure harder to detect as it appears to function correctly mid-process.

DeepSeek quietly forgets instructions, presenting a traditional failure mode.

Claude, conversely, refuses to proceed due to its sensitive safety classifier, while Gemini overthinks tasks to the point of silence, providing no output at all.

GPT 5.5 completes only half of a job before concluding the task is beneath it, demonstrating a unique form of task abandonment.

Some models, such as GPT 5.5, can appear to perform well until the very end of their output, making it crucial for users to understand the precise failure patterns of the model they are dealing with.

Research by Chroma across 18 models demonstrates that accuracy on long inputs can fall by 30% to 50% well before the context window limit is reached, a phenomenon known as context rot.

Coherent, structured text is more susceptible to failure than randomly ordered instructions, indicating that while models may track thousands of instructions, they struggle with clear reasoning when faced with conflicting directives.

While Claude's loud refusals upon failure are annoying, they are easier to detect, making it clear when a task has not been completed.

In contrast, GPT-style partial reports are far more dangerous because they appear confident and polished, yet may silently abandon tasks halfway through, making their output untrustworthy and requiring developers to verify every output.

Running 2,300 queries across seven AI models for this research cost only $29, demonstrating that novel AI research can be highly cost-effective.

Despite the affordability of research-driven testing, production-level monitoring remains essential to proactively identify and prevent silent model failures in real-world applications.

A study titled 'Revisiting the Reliability of Language Models in Instruction Following' tested 46 models, revealing that a model can successfully pass a benchmark but become wildly unreliable if instructions are reworded or reordered.

The exact optimal order of instructions remains an open area of research, as subtle changes can radically alter how well a model follows directives.

New benchmarks, including Firebench, CCR Bench, and Guidebench, are now focusing on evaluating models' ability to follow messy and complex constraints.

The initial difficulty of fitting extensive content into a model's context window, often referred to as compression, has largely been resolved.

The current critical challenge in AI engineering is verifying that a model has actually performed the task exactly as instructed, a problem that requires robust evaluations rather than merely refining prompts, especially as instruction-following capacities have increased tenfold.