Anthropic's 30,000 Daily Code Changes: The Verification Imperative in AI-Driven Software Factories

During a survey, nearly all audience members indicated their coding agents had opened pull requests within the last week. However, only a small fraction of them reviewed every line of code that was merged.

Manish Kapur highlighted this disparity, pointing to a growing gap between automated merging and the capacity for manual review processes.

CircleCI's analysis of 28 million CI workflows revealed a 59% increase in throughput this year. Despite this acceleration, three out of every ten attempts to merge code into production are failing.

Manish Kapur stated that this trend is consistent with what he observes among most of Sonar's customers.

Addy Osmani described two types of software factories. A “lights-on” factory involves agents assisting with tasks such as planning, building, checking, and shipping code, but humans still apply their judgment and taste.

In contrast, a “dark software factory” is more akin to a robotic car factory, where agents autonomously handle the entire process from scoping to shipping. Humans primarily intervene only for the most significant exceptions. Most teams operate somewhere on this spectrum, aiming to increase automation while maintaining trust in the process.

Manish Kapur noted that AI coding is now a widespread practice, with 85% of organizations currently using coding agents, a number expected to reach 100% soon. However, 76% of developers report that AI produces code faster than they can verify it, leading to increased time spent on code review.

Engineering leaders are prioritizing faster shipping while maintaining confidence in code quality. Addy Osmani explained that AI is now used to generate tests, benchmarks, and specifications, making code verification more cost-effective.

For AI agents to produce high-quality output, it is crucial to provide them with methods to verify the correctness and completeness of their work. Manish Kapur stated that implementing such verification can significantly improve outcome quality by two to three times.

The concept of code confidence is considered a scarce resource that must be actively cultivated, especially now that the core coding problem is largely addressed.

Anthropic has dramatically increased its daily merged code changes from approximately 2,500 in May to 30,000. This scaling was critical for projects like the recent refresh of Claude.ai, which saw a three-fold performance gain from merging around 3,000 pull requests in a short period.

The effort involved contributions from various functions, including engineers and product designers, underscoring a comprehensive approach to workflow and verification.

Anthropic's verification workflow begins with Claude performing a first-pass review of the code. A key aspect of this process is ensuring Claude can simulate a real-user experience, which involves end-to-end tasks, browser checks, tool execution, and linting to build confidence in the code.

Addy Osmani explained that if bugs or functional gaps are found, automated hooks run on edits, study the diffs, and enable rapid corrections of pull requests. Teams are also encouraged to codify best practices in shared documentation like Claude.md or agents.md for agent consultation during reviews.

Developers remain responsible for reviewing the verification evidence, such as screenshots or videos, provided by the agent to confirm that the code works as intended across different experiences, like mobile or desktop.

Manish Kapur cited findings from Sonar's upcoming State of Code survey, which includes responses from over a thousand developers. While 84% of developers believe AI features accelerate the delivery of features and fixes to production, pull requests are becoming larger and more complex.

A concerning statistic from the survey reveals that 31% more code is reaching production without manual review. This poses a significant risk, particularly for regulated industries and mission-critical applications where absolute verification and confidence in shipped code are essential.

Manish Kapur shared insights from Sonar's upcoming State of Code survey, which surveyed over 1,000 developers. The survey found that 84% of developers believe AI features accelerate the delivery of features and fixes to production.

Despite this increased speed, pull requests are growing larger and more complex. Furthermore, 31% more code is reaching production without review, which is a particular concern for regulated and mission-critical applications that require absolute confidence in shipped code.

Addy Osmani emphasized that future advancements in software development will stem from a combination of perfecting verification processes and ensuring high model quality. He noted that models like Opus 55 have already achieved a high level of quality.

The critical next step is to share best practices for verification, moving beyond general advice to specify useful types of tests. Addy suggested leaning on end-to-end tests, property tests, and visual verification to build the confidence needed for automatic code merges.

Manish Kapur highlighted that only 5% of companies currently utilize software factories, which he defines as automated loops at scale. However, this is projected to grow to 34% overall, with an even higher adoption rate of 39% expected within the financial services sector.

Addy Osmani reiterated that software factories exist on a spectrum, from “light” factories where humans are heavily involved, to “dark” factories that are fully autonomous. The goal is to progress towards greater automation by enabling agents to generate verification and instill confidence.

Manish Kapur stated that Sonar firmly believes every company should adopt a zero-trust, multi-layered verification approach, with verification integrated into every loop of the software development process. This includes agentic loops, CI verification loops, and code maintenance.

Sonar's approach involves distinct layers for different types of defects. The first layer is deterministic, catching issues like SQL injection and buffer overflows. The second layer uses agentic review to identify logic and intent issues that deterministic layers might miss. The third layer focuses on software supply chain security, detecting CVEs, malware, and malicious packages.

Addy Osmani noted that many enterprises are concerned about the proliferation of agents, often finding hundreds or thousands operating without clear understanding of their actions. These agents can interact with all systems, access data, and even deploy to production, raising significant worries.

Key concerns for companies include agent identity, telemetry for tracking activities, control mechanisms, permissions, and ensuring sandbox integrity to prevent agents from performing unauthorized actions. Manish Kapur added that Sonar addresses this with “guide rails,” which actively direct agents along designated paths rather than merely preventing them from going off track.

The panel discussed the timeline for adopting “dark” software factories, which are fully automated environments. Addy Osmani predicted a transition to “light” factories with some “dark corners” rather than complete automation by the end of 2027.

Manish Kapur suggested Anthropic could move towards dark factories as early as this year or next. The consensus indicated that total automation is a gradual progression rather than an immediate industry standard.

Addy Osmani believes that labeling AI-generated code in pull requests (PRs) is beneficial. He explained that it helps identify constraints and necessary additional checks for code produced by agents. Anthropic, he noted, focuses more on outcomes and is comfortable with Claude being credited as the author.

Manish Kapur supported mandatory labeling in the short term, specifically for the next 6-9 months, emphasizing its importance for auditability in sectors like financial services. Both speakers agreed that regardless of who or what writes the code, human accountability for the outcomes remains non-negotiable.

Addy highlighted that while an agent might initiate work, the human who kicked it off is still responsible for fixing any issues that arise.

Addy Osmani advised leaders to encourage their teams to deeply consider what verification truly entails, focusing on existing testing approaches and identifying gaps that prevent agents from independently signing off on code quality.

For junior developers, he recommended gaining extensive experience in various testing methodologies, such as end-to-end and property tests. The ultimate goal is to achieve a level of comfort where agents can verify code quality independently before merging.

The panel concluded by outlining three core principles for maintaining development speed without sacrificing safety. First, code generation is no longer the primary bottleneck in the development process, as it is largely a solved problem.

Second, strong review and independent, automated verification must be integrated into every loop of the software factory to ensure quality. As Manish Kapur stated, “Fast to write is not fast to ship.”

Third, autonomy in software factories is earned through trust; teams must rigorously verify everything to ensure that code does not incorrectly escape the sandbox before reaching production.