How to Build an AI Code Review Agent: Lessons from Greptile's Daksh Gupta

Gupta introduces himself as a co-founder of Greptile and says the company builds agents that review and test pull requests with the full context of the codebase. By reviewing and testing those pull requests, he says, the agents surface bugs, logical errors, and code quality concerns, primarily in AI-generated code.

He says many thousands of companies use Greptile every day and that about 1% of all pull requests on GitHub pass through it, across many kinds of codebases. Every so often, he says, Greptile meets a company for which it is not the right tool because its needs are too specialized. In those cases Greptile tends to help the company build its own agent, free of charge, for two stated reasons: it has specialized knowledge about doing this well, and it learns how to serve similar customers in the future.

The rest of the talk, he says, is what Greptile tells such companies.

Gupta says customers like three things about Greptile, and he will explain how to implement each. The first is codebase context. Customers find that the tool understands their code. He notes that Greptile began as a codebase context company, teaching language models from 2023 to understand very large codebases, which made it good at indexing code.

The second is that Greptile runs the code. It spins up the stack in a fully loaded, dependency-rich sandbox and tests it by clicking around with computer use, including stress-testing API endpoints.

The third is efficient allocation of inference. Greptile picks the right model and the right amount of inference budget for each subtask in the review and validation process, which he says yields considerably more performance per dollar than an internal tool or competitor could achieve.

He says the talk serves two audiences: teams that want to build their own review agent, and teams choosing a review or testing agent who want to understand what separates a good one from a great one.

Gupta imagines a platform team, or a single AI-enthusiastic engineer, tasked with building the company's review agent. Companies are writing a lot of code, he says, including code produced by vibe coding from engineers and even marketing staff, so the bottleneck has moved to review and validation.

The first step he recommends is to get a sandbox. Many vendors sell them, or teams can use Firecracker. Inside the sandbox goes an agent. He personally recommends a model-independent harness, for reasons he says he will get into, such as Open Code or Pi, both open source. The team then connects GitHub or GitLab, sets up a PR webhook that sends the contents of each newly opened pull request to the agent, and gives the agent a prompt asking it to review the code and send back feedback, which is wired back through GitHub's review API.

Gupta says many people think this is all it takes. It does work in a basic sense: a pull request goes live, its contents reach the agent, and about five minutes later feedback appears. But he lists several things that go wrong. The agent misses bugs, and because bugs are, in his words, fractally infinite, it keeps finding new ones on the same PR. It misses context and misunderstands how parts of the codebase, or its libraries, work. He jokes that companies that never recovered from the microservices sprawl of the 2010s have thousands of repos that the agent would need to understand.

Finally, he says, it gets very expensive, which surprises people.

Gupta says people who have tried this approach report spending between $3 and as much as $25 per pull request on reviewing code with coding agents. He notes that these token costs can accumulate very quickly and that they grow quietly until the bill becomes large.

The fourth problem, Gupta says, is nitpicky comments. He defines them as comments that are technically correct but overly pedantic. They cause two kinds of trouble. For a human reader they are very annoying, like a boy who cries wolf: reviewers must read through a flood of comments and work out which are minor stylistic complaints and which are catastrophic problems.

The second trouble is that nitpicks can wreck the codebase. Review agents typically leave comments on GitHub, and a coding agent picks them up and resolves them. Coding agents are not discerning, he says. They address every comment, and the result is extremely, painfully defensive code, which he says coding agents are particularly good at writing.

The lowest-hanging fruit, Gupta says, is to optimize the prompt. A team typically starts with a very simple prompt, such as reviewing the PR and outputting comments in GitHub's API schema format. He says the prompt should be oriented around two themes: look for specific bug types, and skip the nitpicks.

To find the specific bug types, teams can do some internal data science on existing outages and incidents in their codebase to see which kinds of bugs they are most prone to produce. Those types can be cataloged and the prompt told to look for them. On nitpicks, the prompt should expressly say what a nitpick is and tell the agent not to look for them. He gives examples such as asking why there aren't three layers of defense on something or why error handling is done in a slightly shoddy way.

He warns that a small change to the prompt can have a weirdly large effect on the number of comments the agent writes. This is scary, he says, because you do not know whether it is now missing a bug, since you cannot see the counterfactual.

Gupta explains that review agents, like any reviewer or tester, can tell you about the presence of bugs but cannot tell you about their absence. That is a limitation of the format. The problem is made harder because the most serious bugs are extraordinarily rare and getting rarer. According to Greptile's data science, he says, only about one in every thousand pull requests contains a generally catastrophic bug, or perhaps one in every thousand lines of code. Because such bugs are so rare, a team may release an agent it believes is very good that secretly misses a catastrophic bug, and it may not find out for a long time.

The second improvement, therefore, is a dataset of historical PRs. This is relatively easy for companies that catalog every incident, because they will have a list of pull requests that caused outages in the past. The team can run its review agent over those and measure how often it catches those bugs. This lets the team tune the prompt more confidently.

But bugs change over time. Their topology and ontology shift as the world changes, so the dataset must be updated in real time. Gupta says this is tricky, since unless a team has millions of pull requests, it is hard to anticipate new kinds of bugs. Greptile has an advantage here: it has code from many companies and can test across a very large corpus of pull requests, which lets it improve its agent confidently and gain small margins of performance that internal agents cannot. He says teams can still get reasonably far with their own internal data.

The next improvement, Gupta says, is model inversion. He says he discussed the ways coding agents fail in an earlier talk at AI Engineer in San Francisco, where he found that the types of bugs produced by Claude models and Codex models differ considerably. As an example, he says Claude models are something like three times more likely to produce one-off errors than GPT models, while GPT models are significantly more likely to produce auth bypasses than Claude models. He understands these differences to be artifacts of post-training.

To prevent the consequences, he says, using the same model to review code written by the same model catches relatively fewer bugs, because the model fails in the same ways during review as it did during generation. The solution is to invert the model: if Claude Code wrote the code, review it with a GPT model, and vice versa.

He says that this is straightforward for about 80% of cases and quite tricky for the remaining 20%. Coding agents often leave a fingerprint, such as Codex naming branches in a codex/branch-name format or Claude adding a co-authored-by line to the PR description and commit message. In roughly 80% of cases this identifies the model. In the remaining cases, such as manually created branch names, it is much harder, and teams must develop heuristics, maintain internal catalogs of what agents engineers use, or rely on telemetry from developer machines.

Gupta says Greptile found a 10% higher catch rate when it flipped models. He says this may sound small, but in terms of incidents and outages, 10% is quite a lot. Put another way, there is a 10% higher chance that something will go wrong without model inversion, which can be catastrophic. He strongly recommends using some form of model inversion in the process.

The third improvement is codebase context, which Gupta says he cares about a great deal. Before Greptile did code validation, it worked on codebase context, trying to find better ways to run retrieval over very large codebases. Many people still consider grep to have effectively solved codebase context, but he disagrees for two reasons: grep is extremely imprecise, and it is very expensive.

To illustrate the scale, he says single-digit percentage points of GDP growth last year came from inference, and a very large share of that inference went to coding agents. Looking at a coding agent trace, he says, the vast majority of tokens fall into the same few operations: grep, file reads, and cache writes of file contents. He argues that a non-trivial share of GDP growth is therefore spent on grep, file reads, and cache writes. He also suggests that companies paid by the token may have adverse incentives to find ways to burn more tokens.

The alternative is more surgical methods for deciding where to read. Greptile produces an internal knowledge base in which it checks the entire codebase, catalogs its different parts, and indexes them: for example, that a given area is authentication, how it works, the flows involved, a diagram of those flows, and every file involved.

Codebase context matters for detecting something like an auth bypass. Gupta says it is very difficult to tell whether an auth bypass has occurred, because you need a concrete understanding of whether you are in an authenticated environment when you are in that part of the codebase.

Gupta notes that he has focused on bugs, but code can be very bad even when it is not buggy. It can be of terrible quality, and it can fail to fulfill the expressed intent of the author.

He says he recently saw a post claiming that people increasingly use Slack as their IDE. He finds that a little funny, but notes that a Slack message has little expressed intent: you might say a sentence or two, if that, to ask an agent to fix something, leaving a lot of ambiguity. As a result, a coding agent may misinterpret the instruction or find it too ambiguous to know exactly what to do. Correct-looking code can still be quite wrong.

Greptile counters this by giving agents more context. Gupta says the two types of context that matter most are docs and tickets, and the codebase from the indexer or knowledge base. He says teams can get remarkably far by giving the agent access to Jira or Linear and to Confluence, Notion, or wherever their docs live. Much tribal knowledge subconsciously helps engineers write better code, he explains, and when agents lack it they seem not to know what is happening, partly because people underestimate how much knowledge they themselves hold.

Greptile's approach, Gupta says, is to disintermediate all these context sources with a living wiki. He describes it as a world model of how the company works: everything the system knows about the codebase, everything it knows about the application through production telemetry and logs, and everything it knows about the roadmap from documentation and Linear tickets.

The agent uses this wiki to supply context, which makes it more context-aware. It also makes code searches significantly more precise, because it never reads a file it did not need to read.

The next step is autonomous testing. Gupta gives the example of UI changes: an agent might read through a set of changed React files and conclude nothing is wrong, but when the page is rendered it might, for instance, be hidden on an iPhone.

He says this is one of the more technically difficult problems. Greptile builds dependency-rich sandboxes, which are completely configurable virtual machines, and works with customers to spin up their stack inside them. Greptile's agent then uses computer vision to test the application end to end. Gupta says computer vision has improved a great deal in recent GPT and Opus generations, and that for the first time he thinks end-to-end QA is nearly solved.

Two things are required, he says. The team must document all the flows through its application, and it must have a sufficiently simple build script so that the agent can spin up the stack in its environments. Greptile has forward-deployed engineers who help companies with this, which he says is one of the trickier parts, unless the company already has very good dev boxes.

He then describes what the setup needs: a dev box, PR branch dependencies in which the app runs, and a flow map describing, for example, the auth code and the sign-up flow. With that, the agent can test code end to end and use vision models to observe problems, such as a button that technically appears but is hidden on iPhones or sits outside the first fold on an iPhone Duo.

Even with all of this, Gupta says costs remain quite high: a single pull request review can cost anywhere from about $3 to $15 to $20. The next thing to solve for is using cheaper models for some tasks. He says frontier models are still generally more capable than open source models, though the gap is tightening.

Greptile's split is that about 20% of its tokens go to frontier models for the most critical tasks. The long tail of tokens, such as exploratory file reads, goes to a variety of smaller models. As an illustration, he says Greptile uses a small model for finding context, a very lightweight computer-use model for testing user flows, and a frontier model for analyzing bugs, applying the review process, and first submission. In practice it has about 20 subtasks, each with a specific set of models, and a small research team constantly iterates on which models are used for which task.

He says a team can probably get 70 to 80% of the way there by swapping only very simple tasks to very simple models.

Gupta says this is another area where a very large dataset helps. Without a good dataset, a team might happily replace many subagents with smaller, faster models and not realize that the agent has become quite a bit worse at catching bugs. They would find out only when a missed bug causes an outage or incident.

Overactive file reads can be countered, Gupta says, with a cheap model that looks through all the context pieces and determines which should be included. In theory, a decision model could do this, such as one built with Java or a decisions API, to re-rank the different items and decide what should and should not be loaded into the frontier model's cache, which he calls one of the most expensive and scarce resources.

He closes by saying that teams who do all of this may get quite far in building their own review agent. He invites those who build one to get in touch by email, and notes that teams who choose not to build their own can use Greptile's product, which he describes as free in comparison.