TestMu AI Unveils Full Agentic QA Loop

He emphasized that most AI testing demos stop at merely generating a test, neglecting the ongoing process of quality assurance. The complete loop encompasses four critical stages: authoring, collecting evidence, assurance, and maintenance, ensuring a continuous and robust validation process.

This session specifically focused on applying this comprehensive agentic QA loop to mobile applications, addressing the unique challenges and requirements of that environment. By integrating these four stages, the system aims to create a more resilient and reliable testing framework.

Mahajan explained that if an agent detects a change in the actual implementation, it might modify the test itself to match this new implementation, rather than flagging a discrepancy.

This unintended drift between the planned behavior and the actual implementation is a critical concern, which the KCLI (Knowledge-Centric Learning Interface) aims to manage. KCLI focuses on preserving the original context of the test and mitigating such biased updates to ensure that tests truly validate against intended behavior.

The proposed QA loop uses requirements from Product Requirements Documents (PRDs) as the ultimate source of truth to validate software behavior. Mahajan detailed that the system creates test cases directly from these PRD requirements, ensuring a direct link between product design and testing.

This loop rigorously validates whether the final application behavior aligns with the initial requirements defined by the product team. The KCLI ingests PRD documents, generating and executing relevant test scenarios to automatically detect any drift between the documented intent and the actual implementation.

Desktop environments benefit from rich, easily identifiable DOM structures that simplify automation.

In contrast, mobile applications often require complex gestures like swipes and pinches, and their underlying element locators are not as descriptive. AI agents must therefore adapt their interaction strategies, understanding and simulating these mobile-specific constraints to effectively test applications. This requires a more sophisticated approach to identifying elements and executing interactions.

This process involves storing the PRD content as a blob within the local context directory. By doing so, the system prevents AI from redundant processing of the same information. The extracted state is tracked, allowing the AI to efficiently leverage the stored requirements without re-analyzing the original documents each time.

This process ensures that each use case represents a distinct, verifiable component of the application's functionality. The system manages all entity linking internally within the repository, maintaining a cohesive structure for the generated tests. This independent nature allows for modular and efficient testing.

Acceptance criteria are defined as individually verifiable steps within a test, such as confirming a login has successfully completed. These criteria ensure that specific functionalities meet defined conditions.

In contrast, scenarios are a one-to-one mapping with actual runnable tests, encompassing both automation execution and verification. This clear distinction allows for precise test design, where criteria define expected outcomes and scenarios outline the sequence of actions to achieve those outcomes. The system generates tests that integrate both these elements, providing a holistic validation approach.

The AI agent actively involves users in the test flow design process by prompting for specific inputs like credentials and scenario parameters. Users can review and modify the automatically generated logic, ensuring human oversight and customization. The system maintains robust versioning by committing every accepted or rejected change to the context directory. This approach ensures that all modifications are independently revertible, offering flexibility and control in later phases of development and testing.

The AI agent queries the user about the desired starting state for mobile application tests, specifically addressing pre-existing login states. This involves determining whether the test should begin by logging out and logging in again, or if it should assume the application always starts with a fresh login. The system then generates tests that account for both positive and negative scenarios, depending on the specified initial conditions. This ensures comprehensive coverage regardless of the application's starting state.

This format simplifies the test creation process, moving away from complex scripting languages. The KLI then automates these tests based on the plain-text input, abstracting away the technical complexities of automation. TestMD aims to make test automation more accessible and efficient by focusing on natural language objectives. This plain-text approach allows anyone to define test cases without deep programming knowledge.

The system differentiates between initial test generation, which relies on Large Language Models (LLMs), and subsequent test execution. Initial authoring uses LLMs to generate test tasks for the first time, leveraging their advanced understanding to create comprehensive scenarios. Once a test is automated and transformed into a script, it becomes replayable without requiring further LLM interaction for each run. This approach makes replayable tests significantly more cost-efficient by avoiding repeated LLM usage, as the generated script can be executed directly.

The system automatically generates a detailed evidence file after each test run, documenting all session contents and validation reports. This evidence includes LLM-powered root cause analysis (RCA) for failures, providing insights into why tests failed. It also visualizes test coverage based on the original design criteria, showing how much of the intended functionality has been validated. This comprehensive evidence helps teams understand test outcomes, debug issues effectively, and ensure thorough coverage.

The TestMu AI system maintains identical command structures across diverse environments, including iOS and desktop applications, to ensure cross-platform consistency. Mahajan emphasized that the commands for test generation remain the same, regardless of the target platform. This consistency enables developers and QA engineers to use a unified workflow for cross-platform applications, eliminating the need to learn different tools or syntaxes for various operating systems. The standardized approach streamlines the testing process and reduces complexity.

Flaky tests, particularly in mobile environments, typically stem from dynamic locators that change unpredictably. Mahajan explained that the KCLI addresses this issue by not relying on manual locator management and instead incorporating built-in autohealing capabilities. This functionality allows the system to adapt to changes in element locators during test execution, reducing test flakiness and improving reliability. The tool is designed to manage both local and remote test executions, ensuring consistent performance across different testing setups.

Currently, KCLI agents primarily interact with browsers, limiting their direct engagement with other system components. However, Mahajan noted that the platform is expanding to support direct API calls to backend systems and interactions with databases. This will enable more comprehensive testing that extends beyond the UI layer. Furthermore, a new feature is under development to allow agents to run shell scripts, which will significantly enhance their ability to perform system-level tasks and broader integration testing. These ongoing developments aim to broaden the scope of agent capabilities.

Running an entire test suite as a single batch command is often considered expensive and inefficient, according to Mahajan. He advocates for testing only 'impacted areas'—sections of the application affected by recent code changes—rather than the full suite every time. The KCLI leverages changes in requirements to intelligently identify which tests are necessary to run, thereby optimizing testing efforts and reducing execution time. This targeted approach ensures that resources are allocated efficiently, focusing on areas with the highest risk of new defects.