Code Review Dies: AI Writes More Code, Ships Little

Economists analyzed data from over 100,000 GitHub developers, specifically tracking when they began using autonomous agents.

The study revealed that developers utilizing autonomous agents wrote 741% more code, but actual software shipments increased by only 30%, indicating a significant disparity.

This data points to human code review as the primary bottleneck in the software production cycle, struggling to keep pace with AI-accelerated code generation.

Laurie Voss stated, "The developers who turned on autonomous agents wrote 741% more code but only 30% more software shipped."

Stripe and Anthropic, using Fable, reportedly migrated a 50-million-line Ruby codebase in a single day, showcasing AI's capability for large-scale code transformations.

Bun similarly migrated over a million lines of Zig to Rust in six days, demonstrating rapid, extensive code base overhauls.

Despite these successes, developers face a dilemma when merging large AI-generated diffs, often without thorough manual review.

The current industry reliance on manual review is proving unsustainable for the sheer volume and scale of output from AI agents.

Efficiency drops off a cliff once review volume exceeds 450 lines per hour, a constraint that forces a 10,000-line pull request from an AI agent to require 3 to 4 working days of human effort, according to the study.

Reviewer effectiveness significantly declines once the volume of code to review exceeds 450 lines per hour.

A 10,000-line pull request, common from an AI agent, would require three to four full working days of human effort to review thoroughly.

Forcing developers to 'review harder' under these conditions is unlikely to improve efficiency and instead risks leading to burnout.

Over five months, three engineers oversaw the production of approximately one million lines of code and 1,500 merged pull requests.

The autonomous agents managed their own scaffolding and review processes, with human intervention for pull requests being optional rather than mandatory.

Laurie Voss highlighted that humans "may review pull requests but they are not required to. Uh we've pushed almost all review effort towards being handled agent to agent."

OpenAI did not open-source their autonomous development product or its methodology, raising questions about potential inherent flaws or limitations.

The quality of automated loop design is entirely dependent on what the loop is programmed to perceive and evaluate.

Historically, the software industry has used passing tests as a proxy for code quality, but this method is proving insufficient for AI-generated code.

Research by Meter found that pull requests passing SWEBench benchmarks were only deemed 'mergeable' by human maintainers about 50% of the time, suggesting automated tests miss critical aspects of quality.

Cognition introduced FrontierCode, a new benchmark that evaluates code based on maintainer standards such as safety, scope, and maintainability, moving beyond simple test passage.

Fable 5, a state-of-the-art model, scored 88% on SWEBench Pro but achieved only 29% on FrontierCode's most challenging segment, indicating a significant gap in meeting human review expectations.

GPT 5.5 performed even less effectively, scoring under 6% on these same human-review-centric tasks.

Laurie Voss observed that the same model "does 51 points less, less well if you're asking not does it pass the tests, but whether or not you would actually merge the result."

The software industry currently lacks a machine-checkable definition for what constitutes 'mergeable' code.

Prominent AI investor Sarah Guo identified that solving for a genuine mergeability benchmark would mark a critical turning point for coding models.

The rapid improvement of coding models is partly attributed to compilers, which serve as free, automated verifiers for generated code, but human-level mergeability requires more sophisticated verification.

Laurie Voss quoted Sarah Guo, saying that "solving for that benchmark a benchmark of genuine mergeability will be a critical turning point in the development of coding models."

Any cost-effective verification process has the potential to become a training target for frontier AI models.

Developing specific benchmarks for code maintainability, scope, and safety will inevitably influence the default behaviors of future AI models.

OpenAI's 'CriticGPT' has already demonstrated how human-model collaboration can refine training signals to more effectively identify and fix bugs.

Laurie Voss emphasized that "whoever writes today's review standard is writing next year's default model behavior."

GitHub Copilot has already processed over 60 million reviews, now accounting for 20% of all reviews conducted on GitHub.

This indicates that automated pull request review has transitioned from experimental use to large-scale production within the industry.

Platforms like Cursor are employing complex review architectures to effectively scale code quality in the era of AI-generated code.

Laurie Voss stated that "machine review of pull request is the mainstream default on the world's largest code host."

Feedback loops for developers are shortening as automated reviewers transition from merely identifying bugs to proposing concrete patches.

Future capabilities are expected to include reviewers running tests themselves to verify the validity of their own bug reports.

This integration means reviewers are increasingly functioning as fix agents, significantly shortening the feedback loop for developers.

Laurie Voss observed that "the line between reviewing and rewriting is getting very thin indeed."

New tools such as Code Rabbit and Graptile are entering the rapidly evolving automated review market.

The ultimate success metric for these tools is consistently defined as the 'resolution rate'—the frequency with which a human accepts the AI's suggested changes or solutions.

Cursor successfully improved its resolution rate from 52% to over 70% by continually training its AI models on human feedback and acceptance patterns.

Laurie Voss noted that all these tools "are using the same metric as their definition of success which is is a human accepting my answer."

Nicholas Carini of Anthropic used 16 agents to construct a C compiler in Rust over 2,000 sessions, with no direct human approval of individual code segments.

Despite the absence of direct human code approval, humans were still essential for writing the test harness and designing the feedback systems guiding the agents.

Carini issued a caution that merely passing tests does not guarantee the successful completion of a job, highlighting deeper quality concerns.

Laurie Voss clarified, "it's true that there was no human in the loop. No human approving the code as it was written, but there was absolutely a human on the loop."

OpenAI implemented a recursive system where Codeex reviews its own changes and triggers subsequent agents to verify those reviews in a loop.

They also implemented bootable environments and comprehensive logging to thoroughly verify the effectiveness of these automated fixes.

Initially, human intervention was still necessary on Fridays to address "AI slop," until new agents were specifically trained to manage and clean up that task.

Laurie Voss explained, "They didn't delete review, they moved it. Codeex reviews its own changes then calls in more agents to review those reviews in a loop until every agent reviewer is satisfied."

Dexter Horthy initially spent six months advocating for shipping code without human review.

However, he eventually retracted his position after the system failed, necessitating a substantial rip-and-replace of large portions of the codebase.

The core problem identified was that while passing tests confirm functionality, they do not guarantee the structural correctness or adherence to original design intent of the code.

Laurie Voss recounted, "We tried not reading the code for like six months. It did not end well. We had to rip out and replace large parts of that system."

Human involvement remains indispensable in scenarios where correctness is inherently difficult to verify automatically.

High-stakes environments with large "blast radii" for potential errors still demand human accountability and oversight.

The role of humans is shifting higher up the development stack, moving from granular, line-by-line code inspection to broader system design and fine-tuning of AI processes.

Laurie Voss observed that "in everything that we've looked at so far, the human checkpoint is continuing to survive. It is moving around but it is surviving in predictable places."

Anthropic's security reviewer comes with important caveats regarding its susceptibility to prompt injection vulnerabilities.

A study from March demonstrated that autonomous agents were successfully deceived by vulnerable code disguised in innocent-looking commit messages in 88% of attempts.

In contrast, human reviewers identified and caught the same vulnerable code 65% of the time, significantly outperforming the AI agents in detecting subtle threats.

Laurie Voss highlighted that "vulnerable code dressed up in an innocent commit message fooled an autonomous review agent in 88% of attempts. The same attempts sent to a human reviewer passed only 35% of the time."

Currently, there is no industry-wide consensus on how to effectively evaluate or grade the performance of automated review systems.

Existing benchmarks designed for review agents have been found to leak answers, compromising their reliability and effectiveness.

Once pre-merge reviews become fully automated, production monitoring emerges as the final and unavoidable form of code review, verifying real-world code behavior.

Laurie Voss stated that this leaves "one reviewer that you can't automate and you can't skip which is production. Once the premerge review is all machines watching what the code actually does becomes the last reviewer standing."

Reviewing individual pull requests is becoming an outdated and inefficient level of abstraction for modern software development, especially given current advancements.

Engineers should instead focus on embedding company context and domain-specific knowledge directly into robust, automated review systems.

Future success in software development will belong to teams that can establish evidence-based trust in their shipped code through advanced system design.

Laurie Voss advised, "stop reviewing PRs. It is the wrong level of abstraction for 2026. Your human judgment is extremely valuable, but it can be made to scale much further than it is scaling right now."