New AI Models Outperform Older Versions

Theo challenges David Kramer's (Zeg) assertion that Fable and Astra do not outperform previous high-reasoning models like Opus and Sonnet. He contends that this viewpoint arises from a lack of proper prompting skills, calling it the "worst take he has ever heard." Theo emphasizes that advanced models offer substantial performance advantages, especially when leveraged with appropriate techniques.

Many users of Fable or Astra are encouraged to try switching back to older models to realize that their tasks perform no differently, a sentiment Theo strongly disagrees with. He highlights that sophisticated models, when correctly utilized, demonstrably enhance task performance and efficiency.

Most AI code review bots fail because they only read code statically without actually executing it. Code often breaks in ways that are not apparent from static analysis alone, necessitating dynamic testing. This limitation renders typical AI code review insufficient for verifying complex software changes effectively.

Sandboxing allows AI agents to test various code paths and verify end-to-end functionality, providing concrete proof of success, such as videos and images, rather than simple text approvals. This approach significantly enhances the reliability and depth of AI-driven code reviews. Theo highlights the importance of such dynamic execution, pointing out that agents understand code better when they can run it.

The number of sandboxes required varies, with some reviews needing none and others needing ten, depending on the complexity of the task. This dynamic process replaces simplistic thumbs-up/down responses with actionable visual proof.

By performing autonomous testing of theories regarding potential bugs, orchestrators increase confidence in shipping code through end-to-end verification. This method ensures thorough and adaptable code analysis, moving beyond static checks to active problem-solving.

Astra possesses unprecedented capabilities, allowing it to perform tasks no other model in history could. Despite its incredible potential, the model exhibits inconsistent performance issues, occasionally making significant errors. This duality means that while Astra can achieve remarkable feats, it can also fail in frustrating ways, reminiscent of earlier, less stable models.

These behavioral quirks can lead users to reasonably revert to models like Sonnet, especially when consistency is prioritized over peak, yet inconsistent, intelligence. The unique nature of Astra's performance means that criticisms about reverting to older models do not apply equally to it as they might to Fable. Astra presents a complex case where its strengths are often matched by its distinct weaknesses.

The optimal prompting approach for AI agents significantly depends on the scale of the codebase being worked on. Developers often tend to overcorrect their prompts, especially in high-stakes environments where errors can be costly. The real-world effectiveness of AI agents, particularly in large companies, hinges on adopting flexible and realistic prompting strategies.

Theo categorizes developers into 'Atlassian devs,' who are often found overcorrecting, and 'realistic ones,' who understand the nuanced effectiveness of AI agents. This distinction highlights the varied approaches to integrating AI into development workflows.

Newer AI models are favored not for their ability to push scientific boundaries but because they are "stupid less often." This increased reliability reduces the need for constant human guidance, especially for routine tasks. The focus has shifted from achieving extreme complexity to ensuring consistent performance and minimizing errors.

Theo explains that this preference for reliability means that developers value models that consistently perform well without constant human intervention. The goal is to reduce the frequency of trivial errors, making the models more practical for everyday use.

To achieve horizontal scaling of AI tasks, developers should avoid micromanaging files within their prompts. Instead, models should be empowered to identify and modify relevant files independently, based on a general description of the problem. This approach automates verification processes, such as simulator testing and build pushes, freeing developers from tedious oversight.

Confirming pull request (PR) success can be done using visual evidence, such as videos, rather than relying solely on text approvals. This method streamlines the development workflow, allowing AI agents to handle more complex and extensive tasks autonomously.

The first metric measures the duration of autonomous operation, while the second assesses the reliability of the output before a human needs to get involved. These metrics are similar but distinct, focusing on different aspects of agent performance.

The overarching goal is to minimize the likelihood of the model making errors or "screwing up." By focusing on these two metrics, developers can better evaluate and improve the overall effectiveness and independence of AI agents in various tasks.

Error frequency in AI models increases as the task scope, or "width," expands, meaning longer and more complex tasks are more prone to errors. When a task becomes wider, involving more operations, the likelihood of encountering an "edge case" where a model like Opus performs poorly significantly rises. This demonstrates that more extensive tasks inherently carry a higher risk of failure.

Frontier models succeed by raising the overall 'floor' of performance, making them less prone to these errors even in wider tasks. This distinction highlights that "smart" and "less dumb" are two different metrics of model quality, with the latter being crucial for long-duration tasks.

With each new generation of AI models, the amount of time it takes for a model to have a 50% chance of making a mistake decreases exponentially, particularly for simple, short tasks. This rapid improvement underscores the significant progress in agent reliability over time. Refining traces and prompts can drastically improve agent success rates, allowing for faster and more accurate task completion.

Theo highlights how iterative adjustments to models and prompts enable instant deployment of pull requests (PRs). This continuous refinement process ensures that agents become increasingly dependable and efficient, reducing the need for extensive human oversight.

While some developers might opt for cheaper models, these can inadvertently become more expensive in the long run. The increased need for human oversight and the time spent correcting errors can outweigh the initial savings, making seemingly cheap models ultimately costlier.

Theo emphasizes that professional time is a valuable commodity, and investing in more capable AI tools can lead to substantial time savings and improved efficiency. This cost-benefit analysis suggests that for serious development work, premium models offer a better return on investment.

Theo argues that his professional time is more valuable than the cost of high-end AI models like Fable or Astra. While some developers believe simpler tools suffice for most web development tasks, investing in premium models can significantly reduce the time spent on both the initiation and completion phases of development. Allowing these advanced models to handle complex tasks earlier in the process results in substantial time savings.

If a developer downgrades to a cheaper model, they must invest more personal time both before the task begins and when issues arise. This trade-off underscores why professional developers find premium models cost-effective despite their higher price point.

Between March and the present, median prompt runtimes for AI agents more than doubled, increasing from 53 seconds to over 2 minutes and 20 seconds. This substantial growth indicates a significant improvement in the models' ability to handle tasks for longer durations. Furthermore, the 95th percentile (P95) runtimes, representing the longer end of task durations, surged from under 7 minutes to over 16 minutes and 20 seconds during the same period.

The introduction of advanced models like Fable and Soul is directly linked to this improvement, causing a nearly twofold jump in the capability to perform long-duration tasks. This data-driven evidence clearly demonstrates the enhanced performance and endurance of modern AI models.

For example, a 5% failure rate over 10 minutes can escalate to a 70% failure rate over an hour when compounded. However, by reducing that initial failure rate from 5% to just 3%, the failure rate over four hours dramatically drops from 70% to 50%.

This mathematical demonstration highlights how seemingly minor tweaks in success probability within short timeframes can drastically extend the total feasible runtime of an agent. Such improvements are critical for developing highly autonomous and dependable AI systems, ensuring they can operate effectively over extended periods without human intervention.

Previous iterations of AI models struggled with maintaining coherency over time and efficiently managing context. Despite innovative approaches like Ralph loops, these models were not adept at compaction, tracking past actions, or leaving reminders of attempted solutions. This deficiency led to immediate failures when tasked with longer durations, making sustained autonomous operation difficult.

Manual specification writing offered some help but proved insufficient for ensuring long-term consistency. Early runs with models like Cursor 2 and 3 consistently broke down when given tasks that extended beyond very short durations, highlighting their inherent limitations in maintaining operational integrity over time.

These models can handle continuous runs of 4 to 8 hours, even if minor issues arise, significantly reducing failure rates compared to earlier periods. This enhanced reliability means that agents are now far more likely to produce good results over longer durations.

The success of modern agents increasingly depends on allowing the model to run for extended periods rather than constantly intervening. This shift towards longer autonomous operation marks a significant advancement in automated workflows, enabling more hands-off development processes.

Experienced developers, including critics like Zeg and David K, often fail to fully utilize modern AI by adhering to outdated prompting and workflow habits, effectively operating as if it were still February. Highly talented leaders, who excelled by meticulously preparing before coding and thoroughly vetting after, struggle to abandon these manual processes. They derive satisfaction from this extra effort, even when AI could automate it more efficiently.

Developers tend to test new models against tasks they have already mastered, inadvertently overlooking the models' advanced capabilities to handle novel or complex problems. This psychological resistance hinders the adoption of truly transformative AI workflows, as they prioritize familiar methods over potentially more efficient, AI-driven approaches.