Software Thinking: The New Quality Engineering

For decades, quality engineering has relied on the deterministic nature of software, where identical inputs consistently produce identical outputs. Every test written under this assumption aimed to verify this binary correctness. However, the advent of generative AI, particularly large language models (LLMs), has fundamentally broken this rule. These systems engage in reasoning and interpretation, leading to variable outcomes from the same prompt, making correctness no longer a simple binary condition.

Traditional testing methods, designed for predictable, rule-based software, are insufficient for AI systems that generate novel responses. The focus is now shifting from merely verifying functional correctness to ensuring the reliability and ethical behavior of AI. This transformation redefines the scope of quality engineering into a broader discipline of AI assurance.

This shift implies that quality assurance professionals must develop new strategies to evaluate systems that can produce different, yet potentially valid, responses based on complex internal processes. The goal is no longer just to find bugs, but to understand and quantify the nuanced behavior of intelligent systems.

Generative AI systems can produce conflicting responses to the same query, highlighting their inherent unpredictability. For example, when asked whether a specific contract clause would hold up in a cross-border dispute, an AI system initially responded affirmatively, stating, "Clause 4.2 is enforceable under the governing law selected."

However, upon a subsequent query with identical input, the same system provided a contradictory answer: "No. Clause 4.2 likely fails. The governing law selection is ambiguous." This demonstrated how the exact same input can yield vastly different, even opposite, outcomes, challenging the core tenets of traditional software testing.

For instance, when given the instruction, "Delivery within 30 days of order confirmation," an LLM model responded with "delivery within 30 days with a full refund guaranteed for any delay."

This output appeared correct and confident, but it added an unauthorized guarantee that was not part of the original instruction. Such hallucinations highlight a critical risk: the system's confidence in its response does not equate to its factual or programmatic correctness.

For example, an initial input to Agent A, "Ship this assignment by Friday," was passed to Agent B.

Agent B interpreted it as "Ship it with a high priority." This altered instruction was then passed to Agent C, which further escalated the directive to "Ship immediately with all right checks." This chain illustrates how critical information can be distorted or exaggerated as it moves through multiple agents, leading to unintended and potentially harmful outcomes.

Traditional binary correctness is no longer an adequate measure given AI's probabilistic nature and unique failure modes, which include message drift, inherent biases, and privacy concerns.

Quality assurance professionals must now adopt new metrics and evaluation methods to manage these probabilistic outcomes. The focus is on ensuring that responses from generative AI systems are reliable and aligned with user expectations, even if they are not strictly identical every time. This new paradigm requires a deeper understanding of how AI systems reason and adapt.

These pillars provide a comprehensive framework for assessing the trustworthiness of AI.

An AI system is considered to have failed the trust test if even one of these attributes is not adequately met. These attributes are crucial for validating whether an AI system can be deemed trustworthy and ready for deployment.

Testing AI systems necessitates a structured four-step framework, starting with comprehensive risk identification. For each of the defined trust attributes, engineers must identify specific ways in which the AI system could fail or perform unreliably.

Following risk identification, the next step involves defining quantifiable metrics to measure abstract concepts like hallucination rates. This leads to establishing clear thresholds that serve as pass/fail criteria, similar to traditional testing but adapted for probabilistic AI outcomes. Finally, the framework requires rigorous testing and the collection of evidence to document performance and ensure auditability, providing a clear trail for accountability and continuous improvement.

Quantifying AI risks through specific metrics is one of the most challenging aspects of AI testing, yet it is essential for establishing reliability benchmarks. QualiZeal has already defined over 170 metrics, with continuous development to encompass various risk exposures.

A key example is the hallucination rate, which measures how often an AI system generates incorrect but confidently presented information. This metric is calculated as the ratio of hallucinatory responses to the total number of queries, providing a concrete measure of an AI system's factual accuracy. Such metrics are crucial for objectively evaluating and improving AI trust attributes.

Even if an AI system performs well on one attribute, it is not considered ready for production if it fails any of the others. The system must demonstrate proficiency across all seven trust attributes, including reliability, safety, explainability, security, privacy, faithfulness, and human control.

This comprehensive approach ensures that all identified risks and metrics meet or exceed their defined thresholds. Only when an AI system successfully clears every single criterion can it be deemed fully trustworthy and suitable for real-world deployment.

Instead, quality engineers must adopt a pragmatic approach by defining specific trust score thresholds for production deployment. For instance, a system might be deemed ready for production if its final trust score exceeds 95%.

This paradigm shift necessitates tracking multiple trust attributes, risks, and metrics simultaneously. It acknowledges the inherent probabilistic nature of AI and sets realistic, measurable goals for system trustworthiness, enabling organizations to deploy AI applications with an acceptable level of confidence.

This involves verifying that the AI fetches information exclusively from its designated repository and does not generate novel, unverified content. The fact-checking component becomes paramount to validate the source of every piece of information provided.

Citation sufficiency measures how often a system provides a reference for its claims, while citation accuracy determines if that reference is contextually relevant and factually grounded. Therefore, comprehensive testing must prove that the RAG system consistently retrieves information from its specified knowledge base rather than fabricating it, making citation an essential risk and metric.

For each defined trust attribute, quality engineers must first identify potential risks, then establish specific metrics to quantify those risks.

Only after all metrics have successfully passed the predefined threshold tests can a trust certificate be issued for the system. This rigorous process ensures that every aspect of AI trust is systematically evaluated and validated before deployment.

Generative AI systems inherently possess a degree of creative license, which can be beneficial in certain applications. However, excessive creativity can lead to outputs that deviate from truthful information and become fictional, despite appearing plausible. The challenge lies in defining the precise boundary beyond which an AI's creative interpretation crosses into outright fiction.

Currently, there is no industry standard or clear consensus within the quality engineering community on where this "thin line" should be drawn. Establishing this demarcation is an open and critical question for the future of AI quality assurance.

Risk managers often play a crucial role in defining these parameters, but new specialized AI roles may also be necessary to address the unique complexities of AI systems.

These thresholds, which include confidence levels, must be rigorously established through iterative testing. This ensures that AI agents make reliable and appropriate tool selections based on predefined criteria and validated performance.

When defining a test strategy for AI, it is crucial to first examine the system's architecture to understand its component interactions. This comprehensive approach ensures that trustability is built from the ground up.

Key internal layers, such as retriever accuracy and context generation, are critical points for evaluation. The overall trustability of the AI system is a direct summation of how trustworthy each individual layer and component performs. Therefore, a thorough testing strategy must delve into the granular details of the internal workings to identify and mitigate potential failures.

It is inaccurate to assume that any single attribute, such as correctness or confidence, is universally the most critical. Instead, the weighting of attributes must be customized based on specific application requirements.

For example, bias mitigation becomes paramount for AI tools used in recruitment or those impacting demographic groups, where fairness is a primary concern. Quality engineering frameworks should be flexible, allowing users to assign different weights to specific attributes to align with their particular operational needs and ethical considerations.