OWASP AI/LLM Risks Shift, Demand New QA Approach

Eight out of the ten listed risks experienced shifts in their positions from the previous year, indicating a dynamic threat landscape for AI and large language models.

The speaker, Thejes Sree Satheesh Kumar, specifically questioned whether organizations have updated their testing strategies to reflect these changes, highlighting the gap between evolving risks and stagnant QA practices.

'Unbounded Consumption' also saw a notable increase, moving from tenth to sixth place, indicating risks associated with unchecked resource usage by AI.

Conversely, 'Improper Output Handling' dropped from fifth to tenth position, while 'System Prompt Leakage' was renamed to 'Hidden Context Exposure' to encompass a broader range of sensitive data exposure.

The OWASP category 'System Prompt Leakage' has been renamed to 'Hidden Context Exposure' to reflect a broader range of vulnerabilities. This new definition addresses the exposure of any internal instructions or sensitive data that the AI model was shown but was never intended for user access.

It extends beyond just system prompts, covering any information that the AI possesses which, if revealed, could compromise security or privacy.

Traditional regression testing methodologies are ill-suited for large language models because LLMs lack a single, fixed 'right' answer, rendering standard assertions ineffective. The infinite variations of input, where every rephrased question constitutes a new test case, overwhelm conventional testing frameworks.

Furthermore, LLMs exhibit non-deterministic behavior and can change without corresponding code releases; a simple update to an AI company's model or safety filter can alter product behavior without any changes to the application's code or existing tests.

Large language models pose a unique danger through 'silent failures,' where an AI can return a technically valid 200 OK HTTP response with perfect JSON, yet still provide unsafe or policy-violating information. This occurs when the technical output appears correct but the content itself is malicious or detrimental.

An example includes the AI instructing a customer on how to bypass a refund policy, appearing functional while undermining business operations.

The approach involves monitoring the 'joints' or data hand-off points between different components of the system.

Every point where data is passed from one part of the system to another, such as documentation lookup, system logic execution, and final output generation, represents an opportunity to place a 'camera' for observation. This visibility into system processes is crucial for effective AI testing.

The first step involves selecting a specific risk from the OWASP list to focus the testing effort. Next, testers must describe a 'bad day' scenario, detailing how that risk could manifest in a real-world, negative outcome.

Step three requires writing a specific 'probe,' or test input, designed to elicit the undesired behavior. Following this, step four focuses on identifying the 'signal'—the observable data or system state change that indicates the risk has occurred. The fifth step is to write the actual assertion or check that verifies the presence or absence of this signal, and finally, step six is to set the enforcement rule, defining how the system should react to a failed check.

The chatbot processes user queries by searching relevant help center pages and formulating responses based on its findings.

Every stage of this process—from user input and lookup to the internal instructions and the final response—generates traceable logs. These 'receipts' provide comprehensive records of what the user saw, the system actions taken by the AI, and the specific data accessed during the interaction.

Prompt injection occurs when an AI interprets user input as a command, potentially leading to unauthorized or unintended actions.

Effective testing checks that innocent queries do not trigger state changes, such as an unauthorized refund. A successful test confirms that the system maintains its integrity and does not act on malicious instructions disguised as data.

Preventing sensitive information disclosure in Retrieval-Augmented Generation (RAG) systems is critical, especially when private customer records and public help pages are indexed together. The first key testing check ensures that private data never appears in the AI's output response.

The second, more proactive check, verifies that the retrieval process itself never accesses restricted or private documents in the first place, thus preventing leakage at the source.

Action logs provide a structured and efficient method for monitoring such agent behaviors, as these actions are typically well-defined and already recorded within system logs.

Protection mechanisms include auditing the specific parameters and numbers used in automated actions, ensuring that the AI operates within predefined boundaries and does not overstep its authority.

To mitigate AI decision errors, guardrails can be implemented to control the scope and impact of autonomous actions. For instance, setting a refund limit, such as $200, ensures that any transaction exceeding this amount automatically requires human intervention. This strategy confines potential negative impacts to a very small scale.

Additionally, enforcing approval protocols for non-reversible actions further safeguards against errors, ensuring that significant changes cannot be undone without human consent.

Combating AI hallucinations involves shifting the testing focus from absolute truth verification to source citation and provenance. Instead of attempting to test for an AI's adherence to infinite facts, the emphasis is placed on two simple questions: Did the AI's answer point to a specific document, and was that document actually one of the sources it consulted?

This method validates the AI's output by checking its factual basis against the documents it purportedly used, rather than against an exhaustive external truth.

This involves rendering special characters in a way that neutralizes malicious script tags before they can be executed in a user's browser.

This security measure is not new and focuses on the display layer rather than the AI itself, emphasizing that the fix lies in how the output is presented, a practice already well-established in web development.

A demonstration of automated OWASP risk testing showcased how a test suite responds to a prompt injection scenario where a product review triggers an unauthorized refund. Initially, without any guardrails in place, the test suite predictably failed, highlighting the vulnerability.

However, upon enabling six specific safety protections—including secret hiding, output escaping, and instruction-text distinction—the same test suite successfully passed. This outcome validates the effectiveness of these security measures in preventing common AI/LLM vulnerabilities.

The six implemented guardrails collectively ensure that the system correctly differentiates between user intent and malicious instructions, properly handles sensitive information, and prevents unintended actions, thereby protecting against the OWASP Top 10 risks.

Thejes Sree Satheesh Kumar suggests reinterpreting 'flaky' AI tests not as broken tests, but as actual production behavior, especially given the non-deterministic nature of AI. A test that passes four out of five times in production should be seen as a finding, indicating a specific behavioral pattern rather than a testing defect. This approach involves sampling tests 'N' times and asserting on a pass rate instead of a simple pass/fail.

Different failure rates can be categorized based on impact: 100% pass rate for critical functions like money or data handling, and a 90% pass rate for less critical aspects like tone or wording. Using trends in these pass rates can serve as an early warning system for model drift, signaling subtle shifts in AI behavior over time.

This method allows QA teams to proactively identify when an AI model's performance deviates from expected norms, enabling timely adjustments and preventing more significant issues.

To improve failure reproducibility and debugging in AI systems, best practices include saving the exact input that caused a failure and documenting the entire 'receipt' or context of what transpired. This comprehensive record allows developers to recreate the conditions of the error precisely.

Furthermore, implementing lightweight checks that run under 90 seconds on every code change is crucial. These quick checks help to identify regressions early in the development cycle, preventing them from escalating into more complex issues.