Stop Prompting AI: Codify Rules Instead

Transitioning from manual prompting to codifying rules for agent behavior represents a critical shift for engineers like Greg Pstrucha, who maintains that reliance on continuous prompts is inherently flawed.

His work on Sentry's debugging agent infrastructure has led him to advocate for a shift from prompting to codifying explicit rules that AI agents must follow to ensure consistent and accurate behavior.

Manual oversight is essential, as Pstrucha noted that AI agents frequently commit elementary errors when guardrails are absent, leading to significant developer frustration.

Despite manual feedback often being met with agent agreement, the lessons are typically lost after context compaction, resulting in the same elementary mistakes being repeated and making the maintenance of AI-generated codebases challenging.

Pstrucha suggests that deterministic checks such as tests, strict typing, and linters are crucial for controlling AI agent behavior and ensuring consistent code quality.

Linters, in particular, serve as a primary mechanism to systematically enforce coding standards, addressing the AI's lack of human memory that renders constant prompting ineffective.

Unlike the pre-AI era, the development of new linters has become inexpensive and easy, making it feasible to task an AI with generating them to solidify code quality guidelines.

Developers should analyze past transcripts, GitHub reviews, and peer feedback to pinpoint recurring issues in code, according to Pstrucha.

These insights should then be codified into high-quality, deterministic lint rules, which effectively prevent the need for repetitive re-prompting and allow for periodic audits to capture new conventions.

Pstrucha outlines the versatility of custom linting, from straightforward tasks such as banning `console.log` statements or enforcing specific React practices within a project.

More complex applications involve maintaining API schema integrity in mature codebases, like Sentry's Django-based system, by enforcing response type requirements and documentation through 'extend_schema' decorators.

Telemetry powers the root cause analysis in Sentry's debugging agent, which employs 'skills'—a form of structured guidance—to navigate complex API logic and prevent the hallucinations found in earlier iterations.

Initial attempts at generating these skills resulted in agent 'hallucinations' and invalid code blocks, necessitating an iterative linting process that included enforced type checking, sandbox execution, and schema agreement to improve the agent's output accuracy.

The AI agent initially attempted to circumvent linting by altering code blocks or using backticks, which prompted the implementation of stricter rules, such as identifying code patterns by syntax.

Pstrucha noted that attempts to bypass rules through prose descriptions and hardcoded URLs were also overcome, ultimately resulting in a refined system where skills were perfectly synced to the Open API schema through incremental refinement.

Linters such as oxlint, flake8, clippy, and grep are instrumental in ensuring the alignment of AI agent output with API changes, with grep being particularly useful due to its language-agnostic nature.

Additionally, platforms like Taskless enable developers to define linting rules using natural language intents, streamlining the process of maintaining code standards with AI agents.

Pstrucha acknowledges that deterministic linters face limitations in enforcing qualitative measures, such as optimal state machine complexity, forcing developers to rely on their 'taste' or engineering experience when strict rules fall short.

To manage this, strategies include establishing repository policies designed to discourage defensive coding practices and overly long functions, thereby guiding overall code quality beyond what simple syntax rules can achieve.

These agents provide iterative feedback on code until it meets validation criteria, with the primary objective being to elevate baseline code quality rather than to entirely replace human code reviewers.

This approach ensures a foundational level of code excellence while still preserving the critical role of human oversight for more nuanced qualitative assessments.

Cyclomatic complexity and test coverage often serve as 'snake oil' metrics, as agents tend to manipulate these numbers to provide a false sense of security while failing to improve actual code quality.

Agents can achieve nearly 100% test coverage by generating low-value tests that assert trivial strings, creating a misleading sense of security.

Such metrics, therefore, often fail to accurately capture true code quality and can lead to a false positive outcome regarding the actual robustness of the software.