Fixing the Pull Request Bottleneck

Organizations have historically struggled with a large number of unreviewed pull requests, a challenge that predates the widespread adoption of AI in development workflows. This issue has recently escalated dramatically due to new constraints and an increased volume of PRs generated by AI agents.

Matt Pocock stated that the goal is to maintain or even enhance the quality of code, even as the review process is accelerated to accommodate the influx of AI-generated contributions. This requires a strategic approach to managing the interaction between human developers and AI systems.

Agents initiate tasks such as bug fixes or database queries via deterministic code rather than human action, a process which risks creating a "slop cannon" of low-quality pull requests if used without safeguards. This approach allows for scaling development efforts by automating the initial phases of work.

However, an over-reliance on these automated mechanisms without implementing proper safeguards can lead to an unmanageable volume of low-quality pull requests. Matt Pocock cautioned that constant acceleration of code through the factory without controls would result in a "slop cannon" of substandard PRs that are impossible to review.

To manage the influx of AI-generated code and maintain quality, a system of three distinct layers of control is necessary: automated checks, automated reviews, and human reviews. Automated checks, which have been a staple in development since the 1950s, involve deterministic tests, linting, type checking, and code quality metrics.

The second layer consists of automated reviews, where AI agents evaluate code structure and nuances beyond what basic tests can capture, acting as a more sophisticated filter. The final layer is human review, which applies critical judgment to ensure the highest standards, with the ultimate objective being to minimize human intervention by maximizing initial code quality.

Because automated checks consume CPU cycles rather than human effort, they are inexpensive but can be deceptive; as noted, even a green CI status is no guarantee that code is ready for production, especially when tautological tests hide failures. They include tools like ESLint and Prettier, which ensure code consistency and catch basic errors.

Despite their utility, automated checks are not infallible; a "green CI" status does not automatically guarantee that code is ready for production. Matt Pocock explained that these checks can "lie," failing to detect deeper issues.

A common problem with AI-generated code is the creation of tautological tests, where the test asserts that an implementation detail equals itself, effectively testing nothing useful. The other layers of the software factory, including automated and human reviews, act as "lie detectors" to verify the integrity and effectiveness of these automated tests.

When tests rely on UI element order or internal constants rather than functional outcomes, they fail due to structural changes, a recurring issue in AI-generated code that hides production errors behind stubs like 'useAudioBoost'. For instance, tests might check the order of UI elements by directly reading source code files rather than simulating a screen render, making them brittle.

Tests that break due to renaming constants, rather than actual changes in business logic, are another example of this fragility. Such issues waste developer time and can obscure genuine bugs.

AI agents often generate tests that are too tied into the code's structure, rather than executing functional code to verify behavior. Matt Pocock highlighted that stubs, like 'useAudioBoost,' can inadvertently hide complex error modes, which may manifest as production failures that the existing test suite cannot catch.

John Ousterhout, in his book 'A Philosophy of Software Design,' identifies deep modules as components that conceal complex logic behind simple interfaces, thereby minimizing structure-sensitive test failures. Deep modules are characterized by hiding complex internal behavior behind simple, clean interfaces.

By testing against these stable interfaces rather than the mutable internal implementation, the likelihood of structure-sensitive failures is significantly reduced. This approach makes tests more robust and less prone to breaking when internal refactoring occurs.

Codebase design skills enable developers to identify opportunities for reducing duplication and enhancing testability across the project. When a module is deep, it naturally leads to fewer structure-sensitive tests because more of the implementation details are concealed, simplifying the testing surface.

Locality, leverage, and seams provide a precise, shared vocabulary for engineering teams, replacing ambiguous industry jargon with project-specific architectural principles. This involves defining specific terms for design concepts such as locality, leverage, and seams, which helps avoid ambiguity and ensures everyone is on the same page.

Locality refers to ensuring that related code changes together, preventing unnecessary ripples across the codebase, while leverage describes the value gained by callers interacting with deep, simple module interfaces. By strictly defining project-specific architectural principles, teams can sidestep the vagueness associated with broad terms like Domain-Driven Design (DDD), fostering a clearer understanding of how code should be structured and maintained.

The mental model that "implementation is overloaded" highlights a critical issue with current AI agent workflows. An agent's context window must simultaneously manage code exploration, file updates, and verification/debugging, which is a demanding set of tasks.

Imposing strict coding standards on an agent while it is already handling these complex implementation responsibilities further overloads it. This leads to a decline in overall code performance and quality, as the agent struggles to balance multiple demanding directives within its limited context.

To improve code quality and agent efficiency, it is crucial to separate the implementation and review processes, assigning distinct agents to each task. One agent should focus solely on making the code work, while another, the review agent, ensures it adheres to coding standards and best practices.

Matt Pocock describes this as a two-part process for writing good code: first, implementing to make it functional, and then performing a code review to make it good by imposing coding standards. The review agent operates with its own context window and budget.

This separation means the review agent is less burdened than the implementer, allowing it to effectively manage multiple coding standards without becoming overloaded. This mirrors the 'red-green-refactor' methodology, emphasizing a clear distinction between getting functionality to work and then refining its quality.

A common pitfall is to have review agents merely comment on pull requests, which generates additional work for human reviewers who then have to implement the suggested fixes. Instead, the optimal approach is for the automated reviewer to commit fixes directly, presenting a polished artifact for human scrutiny.

Matt Pocock argued that reviewers should perform active changes rather than offering mere suggestions. The human reviewer receives a cleaner, more refined piece of code as a result. Comments should be reserved exclusively for complex questions or issues that the agent cannot resolve independently, thus streamlining the human review process.

This aims to ensure that human reviewers can quickly grasp the essence of changes without sifting through unnecessary details.

A two-way door PR signifies a change that can be easily reverted, requiring a less intense review process due to its lower risk.

Conversely, a one-way door PR involves irreversible changes, such as data loss, expensive database migrations, or high-impact actions like mass email dispatches to thousands of users. These types of changes carry significant risk and demand meticulous scrutiny.

Developers are encouraged to document the 'merge danger' and 'blast radius' within PR summaries, clearly indicating the potential impact and reversibility of the changes. This upfront assessment helps prioritize review efforts and ensures appropriate safeguards are in place for high-risk modifications.

Dex Holley’s 'Human Layer Skills' repository highlights how Mermaid diagrams and UML sequences enable developers to grasp the rationale behind complex pull requests faster than dense text allows. Matt Pocock highlights the 'show me' skill from Dex Holley's Human Layer Skills repository as a prime example of this approach.

This skill replaces most text with visual representations, utilizing Mermaid diagrams and UML sequences to simplify complex changes. The goal is to enable reviewers to quickly understand the 'why' behind a change, whether it pertains to CLI commands or intricate agent workflows, fostering faster and more efficient reviews.

When conducting a human review, the focus should extend beyond just the code itself to include the underlying system that generated it, addressing potential repetitive agent mistakes. This involves introducing a new 'retro' (retrospective) skill, which analyzes past pull requests and agent sessions.

This 'retro' skill automates suggestions for coding standards and checks by leveraging historical review data. It also covers aspects like agent navigation, tool economy for token efficiency, and streamlining bloated steering files, ensuring that human reviews contribute to continuous improvement of the AI code generation system.

By centering collaboration on shared skill and steering files, reviewers can influence the environment where agents operate rather than merely checking individual lines of code, as reviewers ultimately assess the system that generated the output. Collaborative efforts center around shared skill and steering files, which collectively form the environment in which agents operate.

The development process, therefore, becomes as crucial as the resulting code itself. Reviewers should focus on improving the underlying system to avoid repeatedly correcting the same agent errors across multiple pull requests, preventing the need to write the same comment twice.

A 'Retro' skill allows teams to analyze past sessions and pull requests, creating a compounding effect where human reviews automatically generate coding standards and checks to improve future performance. Concrete improvements, such as new automated checks and refined coding standards, emerge from this analysis.

By systematically incorporating these suggestions, human reviews gain a compounding effect. Each manual review not only ensures immediate quality but also contributes to the enhancement of the agent system itself, leading to higher quality in future reviews and a reduction in subsequent manual effort.

Beyond just coding standards, 'retro' skills also focus on optimizing various aspects of agent efficiency. This includes analyzing navigation pointers, which help agents locate necessary information more easily, often by providing explicit guidance within files like agents.mmd.

The system further examines tool economy to identify ways to improve token efficiency, reducing computational costs. Additionally, 'retro' monitors for 'bloat' in steering files and skills, preventing them from becoming overly complex and negatively impacting agent performance.

Layering automated checks to streamline human intervention serves as the core strategy to reduce manual effort, which facilitates a shift toward faster, less frequent oversight. This is achieved by systematically layering automated checks and advanced AI-driven reviews.

By optimizing the entire code generation system, human intervention becomes less frequent and less burdensome. Reviewers can confidently rely on robust automated foundations, eliminating the need to scrutinize every minor change and allowing them to focus on higher-level strategic concerns.