Killing Code Reviews with AI

Ankit Jain from Aviator aims to replace manual code review with an AI trust model.

He proposes a framework based on a five-layer trust model designed to build confidence in code, enabling it to be merged without human manual review.

Code churn has increased by 861%, indicating a massive surge in code production that overwhelms existing review processes.

The incident-to-PR ratio is rising, suggesting that current reviews are becoming less effective at preventing issues, and review wait times have quadrupled, making the review phase a significant bottleneck. Consequently, over 30% of changes are now merged without any formal review whatsoever.

This process creates a configuration where no one truly reads or thoroughly audits the code, making UI-based review for AI-generated code inherently disconnected from meaningful oversight.

Key functions include knowledge sharing, mentorship, and team onboarding, with alignment being the most important aspect of a release process for development teams.

The critical flaw in this approach is the absence of a feedback loop, as specifications are typically written before all requirements are fully identified, necessitating developers to use interactive agents for iterative feedback.

Specifications often lack critical details that only emerge during the implementation phase, yet developers rarely update specs afterwards, viewing it as a static document.

This static approach is problematic with Large Language Models (LLMs) which are non-deterministic, making strict adherence to an initial specification challenging and causing traditional spec-driven development to fall short in daily software cycles.

Developer intent is currently scattered across various sources, including Jira tickets, Product Requirement Documents (PRDs), and, most critically, the prompts used in interactions with AI agents.

The most significant developer decisions are made during the iterative communication with AI, yet current workflows often create a Pull Request (PR) and discard this vital conversation history. Preserving these prompts is therefore essential for understanding the reasoning behind the final code.

Developers frequently encounter the same types of bugs during manual review, leading to repetitive feedback.

The AI Slop Registry addresses this by allowing teams to codify these recurring issues into automated guardrails, functioning as a training layer on top of standard LLMs that improves continuously through human feedback. Each 'slop' comment effectively trains the system to prevent specific issues from requiring manual review again.

The proposed system involves capturing user-agent interaction sessions to form formal acceptance criteria, which are then combined with the AI Slop Registry to generate an end-to-end test plan.

This verification system automatically spins up a preview environment to ensure the code functions as intended, rather than merely appearing correct.

The review surface thus shifts from scrutinizing line-by-line code differences to evaluating evidence that the developer's intent aligns with the final outcome and meets acceptance criteria.

Large Language Models (LLMs) can automate the often tedious process of creating and maintaining comprehensive test plans, which consist of specific criteria and invariants.

Reviewing the test plan, rather than raw code, allows for human input on high-level governance and aligns with behavior-driven development principles, inviting non-developers like Product Managers and designers to contribute effectively.

Since not all systems can be 100% deterministic, a hybrid approach incorporates Large Language Models (LLMs) as a fallback mechanism for verification.

AI agents can browse applications and capture visual evidence, such as screenshots, while database snapshots, combined with this visual evidence, provide robust proof of functional changes.

This method of reviewing intent and collected evidence prevents the common failure of using the same AI to both write the code and generate its tests, ensuring independent verification.

To eliminate repetitive manual feedback and focus on team alignment, teams can codify recurring comments from their last 1,000 reviews into an 'AI Slop Registry'.

While the initial setup of this registry requires effort, it pays dividends by maintaining semantic accuracy at scale and allowing teams to focus on overall team alignment rather than just code syntax.