AI Code: Fluent But Not Understanding

The session aimed to explore the true requirements of trust and the necessary governance frameworks, introducing Neelmani Verma, an Industry Principal at Infosys, who leads the consulting group within Infosys Quality Engineering.

Neelmani Verma began her presentation with a confession, revealing that all the examples she would demonstrate were built using an AI coding agent, and approximately one-third of the initial output was subtly incorrect. She clarified that AI does not write inherently bad code, but rather code that often appears better than what a tired human engineer might produce late in the day. This immediate observation underscored the core challenge: AI code is fluent but frequently lacks genuine understanding of human intent, making errors difficult to detect.

Verma illustrated the problem with a scenario involving a four-line AI-generated function that looked correct, with sensible variable names and a numerical return. All accompanying tests passed. However, the function subtly violated a core business rule: loyalty rates should apply to the pre-tax subtotal, not shipping or tax. The AI function applied the discount to the entire order total, meaning if a shipping fee was later added, the discount would silently begin applying to it, leading to incorrect calculations and potential financial discrepancies.

The AI-generated code presented its incorrect logic with the same confidence as a correct version, highlighting a fundamental shift in the paradigm of code quality. Traditional quality engineering has relied on the implicit assumption that the person writing the code understands its underlying intent. This assumption forms the basis of the entire development lifecycle, from initial writing and review to testing and final merge. AI-generated code, while not violating the mechanical steps of this chain, fundamentally breaks the underlying assumption of intent, making its errors harder to notice because the development process appears to be functioning correctly.

First, it is fluent but lacks true understanding; for instance, an AI agent once generated a banking reconciliation script that performed correct arithmetic but applied the wrong currency conversion rule, passing all unit tests because they checked arithmetic, not the conversion rule. Second, the underlying model for AI code is unstable; unlike human developers whose judgment is relatively consistent, AI models can change, making previous testing irrelevant without clear indicators. Third, trust in AI code decays after merging due to silent changes in dependencies or data distributions, similar to flaky test environments where tests that once passed reliably begin to fail.

Neelmani Verma elaborated on the third problem: trust in AI code decays after merging, a phenomenon common in flaky test environments where tests fail due to silent environmental shifts. AI-generated code shares this vulnerability, but the drift is harder to detect as it can stem from dependency schema changes or shifts in real traffic, altering behavior without explicit notification. This necessitates a framework that inherently distrusts code merely because it previously adhered to rules, echoing the principles of zero trust in security. The challenge lies in continuously verifying trustworthiness that is constantly in flux.

The solution is to establish explicit acceptance criteria *before* the agent produces code, preventing an echo chamber where the AI-generated test simply validates the AI-generated code.

This moves away from treating every AI-generated change with the same level of freedom. Just as risk-based testing applies varying depths of testing to human-written code—e.g., a marketing banner change versus an interest rate calculation—AI-generated code also requires this risk-based lens. The decision on where to draw these lines is a crucial consulting and engineering task, often falling to the Quality Engineering (QE) partner.

The third pillar mandates that trust in AI code has an expiry date, expiring on a schedule or upon detection of a 'drift signal,' rather than being permanently signed off until a bug is filed. This addresses the vulnerability of AI code to environmental changes, such as dependency versions, schema fields, or data distribution shifts that silently alter function behavior. The fourth pillar, 'micro-segmented view,' replaces uniform review processes with a targeted approach. Recognizing that senior engineers' judgment is a scarce resource, it allocates more scrutiny and time to areas with the largest 'blast radius' or highest potential for failure, especially critical in regulated industries requiring strong audit trails.

The fifth pillar emphasizes continuous re-attestation, moving beyond static trust to an ongoing verification model. This is especially vital for regulatory industries where audit trails and traceability matrices require sustained evidence of code trustworthiness, rather than relying on a single merge date. Verma then transitioned to a live demonstration to illustrate these principles in action. The demo focused on the same loyalty discount function discussed earlier, which had undergone the newly described process, showcasing how the framework tackles the 'trust problem' in real-time.

On the left panel, the assertion was explicitly linked to a policy document, confirming that the discount applied only to the pre-tax subtotal. This verification, logged as 96% attested, served as a crucial evidence record, analogous to what an auditor would require for a banking client. The log detailed which AI agent generated the code, who verified it, and critically, when it was due for re-evaluation. This established a baseline for continuous verification, not static trust.

The demonstration advanced to 'Day 12,' revealing a significant change that would not appear in a routine pull request. A silent 'drift' was detected by the system, indicating an environmental shift, even though the code itself remained untouched. This marked the first real hit to the code's trustworthiness. By 'Day 28,' a new field had been silently created, broadening the underlying context. While the assertion itself didn't weaken, the world around it expanded, causing the same formula to work incorrectly. This drift was further compounded by a production data dump that exposed previously unexercised edge cases and boundary conditions.

This automated guardrail immediately prompted a gate response from the system. Upon re-attestation, the new score dropped to 89%, a reduction that was not a bug but a deliberate reflection of the function's history. The model now carries the memory of the two prior drift events, ensuring it receives closer scrutiny in subsequent cycles, similar to a credit history that accumulates past events.

The system ensures that trust is not merely reset after re-attestation. Instead, the model retains the history of drift events in an evidence ledger, ensuring that functions with a history of issues are watched more closely than those with clean records. This establishes a continuous verification loop: 'Never trust, but always be verified.' This approach provides a detailed record at every step, allowing QE teams to track metrics like specification coverage and attestation freshness based on the organization's risk appetite. This historical context is vital for making informed decisions about code trustworthiness.

For QE teams, this new framework necessitates distinct metrics, including specification coverage, attestation freshness, and risk appetite-based assessments. While many organizations have not yet formalized roles for this, it is often the first recommendation for assessing a client's AI QE maturity. The tooling involves drift monitors and attestation ledgers, which become as central to the pipeline as test runners are today, emphasizing an operating model decision. Verma underscored that syntactic correctness is no longer a reliable proxy for semantic correctness; AI-generated code demands a specific QE lens focused on intent alignment. Traditional coverage metrics are insufficient, requiring layered and continuously renewed verification, not just a single pass rate.

The framework requires governance checkpoints and escalation paths within the Software Development Life Cycle (SDLC), along with the same scrutiny given to an unfamiliar vendor's code. This is crucial because AI models introduce continuous changes underneath the surface. The overarching principle is that quality frameworks must be built to be continuously 're-earned,' not merely signed off once and forgotten. This approach has proven effective for clients, creating lasting frameworks. Nikhil Saxena then opened the floor for questions, starting with whether AI-generated code should face a higher quality bar than human code, or the same.

In response to a question about whether AI-generated code should face a higher quality bar, Neelmani Verma stated that it requires different metrics and parameters compared to human-written code. She emphasized the practice of "LLM evaluating LLM" for an independent assessment, which provides a crucial, unbiased evaluation of AI code. When asked about the frequency of framework re-evaluation, Verma explained that it must be dynamically built-in and auto-triggered, without a fixed schedule, but rather based on configurable thresholds that trigger different signals for attention.

However, she observed a significant shift towards more AI-generated code, with the percentage of human-written code dwindling. She concluded that the distinction between human and AI-generated code is becoming less relevant; the focus should instead be on the trustworthiness and quality of the code itself, regardless of its origin. Nikhil Saxena thanked Neelmani Verma and the attendees, concluding the session.