Evidence-Based QA for Regulated Industries

In regulated industries, simply knowing that a test passed marks only the beginning of compliance discussions. According to Tanvir Farukq, teams must provide granular details, including which specific test ran against which build, and furnish proof of its successful execution. This stringent requirement stems from the need for complete traceability, especially when automated agents conduct testing and potential bugs surface months later, demanding a historical context of test performance.

Regulated sectors frequently face challenges due to the lack of detailed historical test data, making it difficult to trace the exact conditions under which a specific test was run. The core issue is proving that a particular software version met all quality standards at the time of release, requiring comprehensive documentation beyond simple pass/fail outcomes.

Despite thousands of automated test runs occurring weekly, a vast amount of critical runtime data is routinely discarded. Anubhav Singhmaar highlighted that while detailed evidence such as screenshots, network logs, and console logs are generated, only basic pass/fail results typically survive the testing process. This practice leads to a significant loss of context, as rich diagnostic information that could be invaluable for debugging or compliance is quietly thrown away.

This data includes crucial artifacts like detailed interaction logs and system states at the moment of testing. The problem is not merely about storage but a systemic approach that prioritizes minimal reporting over comprehensive data retention, leaving teams without the necessary evidence for deep analysis or audit requirements.

TestCLI, developed by TestMu AI, enables users to describe testing requirements in plain English, driving real browsers or mobile applications. As Anubhav Singhmaar explained, this tool can be integrated seamlessly into terminal environments or continuous integration (CI) pipelines, automating the execution of tests based on natural language inputs.

This agentic approach simplifies test creation and execution by abstracting away complex scripting, allowing developers and QA engineers to focus on defining desired behaviors. The system translates these descriptions into executable commands, providing a flexible and powerful way to automate testing across various platforms.

Traditional QA reports provide trend lines tracking performance across multiple builds and detailed environmental data for each test run. Anubhav Singhmaar noted that these reports often meticulously distinguish between product and test defects, offering valuable insights for daily operational work. However, while useful for immediate oversight and identifying general trends, these reports fall short in providing the deep, granular evidence required for compliance in regulated industries.

These reports usually focus on high-level summaries and aggregate statistics rather than raw, immutable evidence of test execution. The underlying data, often critical for forensic analysis or external audits, is frequently summarized or discarded, limiting their utility beyond routine development cycles.

A systemic issue in QA reporting is the significant data reduction that occurs when test results are transmitted between different systems, such as to CI gates or auditors. Anubhav Singhmaar emphasized that while a test runner might capture a screenshot for a failure at 2 AM, this rich context is often stripped down to merely 'failed plus stack trace' when passed to other platforms. The industry's standard vocabulary lacks the necessary terms to convey deep evidence, like network traffic or element interactions, leading to critical information loss.

This loss of context is immediate and pervasive, occurring at the point where detailed test outputs are reduced to simpler formats for inter-system communication. Such a process fundamentally undermines the ability to reconstruct test failures accurately or provide comprehensive proof of quality, leaving gaps in audit trails and making root cause analysis more challenging.

While minimal data like pass/fail bits are inexpensive and easily portable, high-value test data such as browser states, compute logs, and detailed runtime metadata often remain confined within their original testing frameworks. Anubhav Singhmaar noted that the 'expensive thing,' which includes browser usage, compute resources, and wall clock time, rarely leaves the environment where it was generated. This confinement makes it difficult to extract and use this critical information across different systems or for external verification.

Test frameworks typically prioritize their internal storage and reporting mechanisms, inadvertently creating silos for rich test data. This practice prevents a holistic view of test performance and complicates efforts to standardize evidence collection, as the most valuable diagnostic information is not readily accessible or transferable.

Vendor lock-in in QA is not primarily due to the cost of porting tests, which can be done in about a week of work. Instead, it arises from wiring numerous dependencies, such as dashboards, CI gates, and analytics, to the specific outputs of a single testing framework, according to Anubhav Singhmaar. This extensive integration makes switching frameworks expensive and complex, as every interconnected system must be reconfigured.

The lack of an industry-wide standardized schema for test reporting exacerbates this problem. Each framework produces data in its proprietary format, forcing organizations to build custom integrations that create deep dependencies. This fragmented ecosystem makes it difficult for companies to adopt new tools or consolidate their QA operations without incurring significant costs and disruption.

The emergence of AI-generated tests, which can rewrite themselves autonomously, poses a significant challenge to traditional flakiness detection and trend analysis. Anubhav Singhmaar highlighted that if a test quietly changes between builds, historical comparisons become meaningless because the original version no longer exists. This silent mutation undermines the integrity of historical data, making it impossible to reliably compare test performance over time.

Screenshots alone are insufficient to address this problem; a cryptographic hash of the test file is required to ensure its immutability and validate its version. Without a mechanism to guarantee that a test remains consistent across runs, any analysis based on historical data, such as build-to-build comparisons or flakiness detection, loses its foundational validity.

Anubhav Singhmaar pointed out that compliance requires an external party, without access to internal systems, to independently reach the same conclusions. Dashboards cannot meet this standard as their data is often dynamic and subject to change.

Furthermore, CI logs frequently expire, leading to the loss of long-term data essential for audits, and screenshots placed in wikis become detached from their execution context. Retention policies often delete critical evidence before it can be requested for compliance purposes, exacerbating the problem. As compliance requirements grow, current storage capabilities and reporting mechanisms prove inadequate, failing to provide the verifiable and enduring evidence needed.

Anubhav Singhmaar announced that this new format creates one sealed archive for each test run, ensuring data integrity and portability. The structure of an '.evidence' pack includes two main levels: L0, which contains core files, code, and test definitions, and L1, which captures logs, screenshots, and detailed failure records.

Crucially, the system hashes files rather than parsing them, guaranteeing the integrity of the evidence against tampering. Profile rules are also embedded within the format to ensure backward compatibility for many years, making '.evidence' a robust and future-proof solution for verifiable QA. This approach ensures that all necessary data for compliance and debugging is encapsulated in a single, unalterable package.

A key distinction in the '.evidence' approach is treating the 'pack' as the authoritative record, rather than the report, which is merely a rendering of that pack. Anubhav Singhmaar lamented that the industry has historically discarded the record while preserving only the rendering. This new standardization allows consumers to integrate once and read any result, streamlining the N x M integration problem often encountered in QA.

By providing a machine-readable format, '.evidence' enables agents to perform validation via exit codes, making automated compliance checks feasible. The sealed and portable nature of the pack further allows third-party verification without needing direct access to internal systems, significantly enhancing trust and transparency in the QA process.

The TestMu AI team has developed the Evidence CLI as the reference implementation for the '.evidence' format. Anubhav Singhmaar explained that this command-line tool validates a pack against its contract and then seals it, ensuring integrity. The CLI offers three core functionalities: 'Finalize,' which derives totals, hashes definitions, and seals the archive; 'Validate,' which performs a conformance check with exit codes (0 for valid, 1 for invalid); and 'Merge,' which combines multiple packs from sharded CI runners into a single, cohesive artifact.

These tools empower users to manage and verify test evidence effectively, addressing critical needs for data integrity and aggregation. The 'Merge' functionality is particularly important for large-scale testing environments where tests are run in parallel, allowing for a consolidated view of overall quality.

To create a sealed validation pack, each step within the test execution must include a unique ID, a status, and a strictly increasing ordinal number. Anubhav Singhmaar emphasized that the validator will reject any duplicates, ensuring the integrity and proper ordering of the test steps. Additionally, a producer name is required for reader identification, allowing for clear attribution of the test evidence.

This strict adherence to sequential ordering is crucial for the accurate reconstruction of the test run, providing a reliable timeline of events. The detailed metadata within each step ensures that all actions and outcomes are clearly documented, which is essential for audit trails and detailed root cause analysis.

The '.evidence' format contract allows agents to be recorded as the primary author of test records, not just human developers. Anubhav Singhmaar explained that this feature is not a mere demo but a core part of the contract, enabling agents to file bugs and feature requests against the format itself. This agent-centric identity ensures accountability, especially when AI agents write or repair tests autonomously during off-hours.

By formalizing agents as authors, the system recognizes their increasing role in automated testing, providing a clear audit trail for agent-driven modifications. This approach is vital for maintaining transparency and trustworthiness in environments where AI plays a significant role in test development and maintenance.

Anubhav Singhmaar highlighted that the JSON schemas within the open-source project serve as the ultimate authority, superseding any documentation. In cases of disagreement between the prose and the schema, the schema always prevails, guaranteeing a single source of truth.

The project also includes a conformance fixture corpus for producers, allowing them to validate their implementations against known good examples. Decision logs are meticulously maintained to document the reasoning behind key design choices, providing a clear understanding of the format's evolution and rationale for future contributors and users.

TestMu AI delineates between the KCL and Ken CLI, with KCL guaranteeing the shape of the '.evidence' format, and Ken CLI populating its content. Anubhav Singhmaar clarified that the container for the format is open, recognizing that a format without a reader is merely a file extension. The actual engineering value and complexity lie in the data capture process, which Ken CLI handles.

This distinction ensures that while the structure of the evidence artifact is standardized and open, the flexibility for how data is captured and filled remains. This allows various tools and engines to integrate with the '.evidence' format by implementing the capture logic, while still adhering to a universal structural contract.

The '.evidence' standard aims to resolve four fundamental problems that have long plagued QA reporting. Anubhav Singhmaar explained that it replaces unstructured reports with step-specific packs, ensuring that critical information is captured at each stage of a test run. It also unifies disparate reporting frameworks into a single, standardized shape, overcoming fragmentation in the industry.

Furthermore, '.evidence' replaces human-authored test definitions with hashed, machine-readable records, enhancing trustworthiness and preventing silent mutations. Finally, it enables agentic verification through offline, verifiable, and machine-readable logs, allowing for automated and reliable compliance checks without human intervention.

Anubhav Singhmaar stated that the sealed file format allows it to travel independently, meaning that any party checking the evidence needs no account, login, or access to internal systems, whether TestMu AI's or the user's. This self-contained nature significantly simplifies audits and enhances security.

Crucially, test definitions are hashed inside the pack itself, making any post-facto modification immediately detectable. This cryptographic integrity ensures that the evidence cannot be tampered with after creation, providing an immutable record that stands up to the most rigorous compliance requirements and fostering trust in the reported test results.