TestMu AI Unveils Full Agentic QA Loop

Current AI agents frequently write and verify their own tests, creating a self-referential loop that generates a 'false truth.' These tests are often brittle and slow, leading to a situation where they pass in local environments but frequently fail when deployed to production.

Human effort currently serves as the primary driver in converting PRDs into executable test cases, leading to a lack of complete transparency within organizations regarding how business use cases translate into testing.

This loop spans design, test authoring, execution, and evidence collection, allowing a product manager to input a PRD into a single command-line interface (CLI) that then generates acceptance criteria, scenarios, and tests. Tests are authored in plain English, eliminating the need for complex selectors or XPaths.

Attendees are encouraged to follow along on their own machines, utilizing the provided GitHub documentation to implement the concepts in real time. This hands-on approach aims to provide immediate practical experience with the new system.

The AI initially classified Use Case 3 (Managing the Cart) as medium risk. The speaker demonstrated the ability to override this classification via a simple CLI command. The system then updated the use case to a high-risk status based on this user input, showing flexibility in test prioritization.

The CLI provides a `context list` functionality that displays the extracted context and requirements. This command verified the four generated use cases by presenting them under headings that show the original PRD source and the details created around the discussion. This feature offers a clear overview of the system's current state and parsed inputs.

The system automatically handles user approval for design parameters, proceeding with the design phase after confirmation. This streamlined process automates the initial setup of tests.

This capability allows for assertions against underlying technical behavior, providing a more comprehensive validation of web applications. The demonstration effectively used a static application to showcase this automated validation, confirming that the tool examines various layers of application functionality.

The system automatically maps specific acceptance criteria to corresponding test scenarios, performing a crucial 'wiring' step. This process creates automated traceability, linking business intent to test coverage without manual intervention. It also tracks development progress through various phases and monitors credit consumption.

The lineage feature enables tracking of updates if the PRD is re-ingested, maintaining clear traceability.

Tests are generated and understood by the CLI agent in plain English, eliminating the need for code-based locators. Users can manually write test files (test.md) directly in this natural language format. This approach simplifies test creation and makes it accessible to a broader range of team members.

The agent dynamically decides actions based on defined intents to fulfill acceptance criteria. During the initial run, it 'figures out' the flow, while subsequent runs treat it as a replay, streamlining repeated executions.

The agent automatically saves artifacts upon test completion, providing essential evidence of execution. These artifacts are linked back to the internal TestMu AI test manager, ensuring that every test run is verifiable. The system also allows users to replay or view historical execution steps without needing to re-run any code, aiding in debugging and analysis.

Evidence generated by the testing process is hosted on evidence.lambdatest.com but served directly from the local machine, ensuring that nothing is stored on the cloud. This local hosting mechanism prioritizes data privacy and allows the evidence file to be easily shared with others for external review without exposing sensitive data to cloud environments. This approach offers flexibility and control over test results.

Test failures often signal a 'drift,' indicating a mismatch between the Product Requirements Document (PRD) and the actual application implementation. This drift is addressed by re-ingesting the updated feature into the system. The system then identifies which specific tests need to be updated, ensuring that testing remains aligned with the evolving product.

The acceptance criteria, once established, cannot be modified by the execution or authoring agents. This immutability ensures that agents must adhere to the predefined criteria, which must be pre-approved by the user. The system is designed to force a test failure if an agent deviates from these defined criteria, preventing bugs from being silently masked.