AI Transforms Mobile Game QA with Vision-Based Agents

The company's co-founders, who met at university, began their journey with a background in machine learning and gaming. Their early work was heavily inspired by the advancements in gaming AI made by research labs like DeepMind and OpenAI.

Initial technology demonstrations showcased AI playing complex games, tasks previously considered impossible for artificial intelligence. This pioneering work involved developing AI capable of interacting with various game environments.

Through this research, they discovered that the core technology used to create gameplay AI was highly applicable to quality assurance testing for mobile applications. Jan Schnyder explained, "one thing led to another and we discovered essentially that you know one of the main use cases for having AI that can play games is obviously also to test them."

The AI agent functions entirely on a vision-based system, relying on a live stream from the device rather than traditional methods. It completely bypasses accessibility IDs or DOM tree elements, which are often fragile and require constant maintenance.

This approach allows the agent to interact with applications through taps, swipes, clicks, and keyboard/mouse actions, mimicking human behavior. This makes the interface identical for any type of game or application, ensuring broad compatibility and eliminating the need for specific coding.

Current AI technology faces limitations when applied to highly complex and competitive 3D games, such as Overwatch. The AI has not yet achieved superhuman capabilities in these demanding environments.

However, for simpler mobile UI navigation and many mobile games, AI performs comparably or even surpasses human testers. Complex 3D games present a significantly greater challenge for current AI models due to their dynamic environments and intricate strategic requirements.

The integration of AI into QA is not about replacing human testers but rather complementing their roles. Manual QA and AI tools are viewed as synergistic, each bringing unique strengths to the testing process.

QA professionals will transition into roles that focus on the more human aspects of testing, such as playtesting and evaluating user experience and emotional responses. This allows for a more condensed team of experts who can delve into the nuances that AI cannot yet fully grasp.

Humans will be responsible for instructing and observing an 'army' of AI agents, orchestrating their tasks and interpreting their findings to optimize the testing workflow. Jan Schnyder emphasized, "you can now essentially have a very condensed team of like QA professionals who can focus on, you know, the important or more human aspect of like the play testing, the feeling and stuff like this while they can control an army of these agents."

AI can effectively evaluate technical roadblocks and confirm the 'completablity' of features within an application. This includes identifying bugs that prevent a user from progressing through a defined flow.

However, measuring subjective aspects like 'fun' or a user's dopamine response remains largely beyond the current capabilities of AI. Analytics gathered from actual players consistently provide more valuable insights into these experiential qualities than AI-generated predictive data.

AI agents can be instructed using natural language, allowing human testers to convey specific knowledge and testing objectives in an intuitive manner. This capability enables a more seamless transfer of human expertise to the AI.

Test design files can be shared directly with the AI, guiding the execution of test cases and ensuring alignment with project requirements. Despite these advancements, human experience remains crucial for effectively guiding the AI's learning curve and optimizing its performance.

Verification testing primarily focuses on scripted regression tests, designed to identify known bugs or issues within established application flows.

Explorative testing, a newer application of AI, allows agents to discover new and unexpected bugs without predefined test paths. This autonomous exploration can uncover issues that might be missed by traditional scripted approaches.

However, human verification remains essential for these AI findings. Testers must still review the AI's discoveries to differentiate between actual bugs and intended application behavior, ensuring accuracy and relevance.

The shift to AI-driven QA eliminates the need for technical expertise in scripting, which was a significant barrier in traditional automation. This lowers the entry barrier for mobile QA automation, allowing more team members to participate.

AI agents are inherently smart and adaptable to various situations, unlike brittle, script-based automation. They can automatically adjust to changes in the UI or environment without requiring constant maintenance work.

Justin Yuan noted, "If you compare it to traditional automation, you now with AI do not need to be technical to be able to do mobile QA automation. With AI agents since they're smart, they're adaptable to different situations. So if there's any changes in the environment, in the menu, they can actually adapt to this without needing any kind of maintenance work."

AI agents are finding unexpected applications beyond traditional testing, including competitor analysis and compliance monitoring. Companies are deploying 'armies' of these agents to run on competitor apps, gathering insights into their offerings and strategies.

This automation of manual workflows extends beyond testing, impacting areas like social media monitoring and compliance enforcement. Agents can automatically collect and process data, providing valuable market intelligence without requiring manual scripting for each task.

High-profile fines, often reaching millions of dollars, are increasingly capturing the attention of executives, making compliance a critical concern. This financial pressure is driving a fundamental shift in how compliance is perceived within organizations.

Compliance is no longer merely a cost-reduction exercise but is transitioning to a core focus in product design. Jan Schnyder noted, "I think going forward there's going to be a lot more attention given to uh compliance as a not necessarily as a uh you know a cost reduction problem but front and center for product design."

The black-box testing approach prioritizes mimicking the experience of a human user as closely as possible. This involves the AI agent looking at the device interface, similar to how a human would, rather than injecting itself into the application's code.

This method is highly effective at catching visual issues, such as invisible but interactive buttons, which code-level injection might miss. Bugs like these are classic examples of defects best detected through visual inspection.

Furthermore, the integration-free setup allows teams to initiate testing immediately by simply providing build files or URLs, streamlining the QA process.

Verification testing involves targeted, often human-in-the-loop workflows designed to check specific requirements and known test cases.

Discovery testing, on the other hand, allows for autonomous exploration of applications without predefined test cases. This mode enables the AI to uncover unexpected behaviors or bugs through unscripted interaction, broadening test coverage.

Managing evolving test suites, especially to avoid redundancy when generating new tests, remains a significant challenge. Current solutions heavily rely on a human-in-the-loop approach to curate and maintain these requirements.

QA managers play a crucial role in overseeing the test landscape, determining which tests already exist and what new requirements need to be addressed. AI facilitates the speed of test creation, but human oversight is critical to prevent the generation of redundant tests.

Nicolas Muntwyler explained, "for now this is still uh very much with a human in the loop so we basically have humans using our tool and basically be the QA manager that like sees what kind of tests already exist list, what are the new requirements?"

While AI agents significantly enhance testing capabilities, they may still miss specific execution paths or nuanced scenarios. Human reviewers are vital in identifying these overlooked areas, ensuring comprehensive test coverage.

Humans define the intent for the AI to follow, guiding its exploration and verification processes. This symbiotic relationship ensures that the AI's efforts are aligned with the overall testing strategy.

This approach allows for a robust QA process where AI handles repetitive tasks, and humans apply critical thinking and domain expertise.

AI technology still struggles with assessing subjective qualities such as whether an experience is enjoyable or 'fun.' These aspects are inherently matters of taste and human perception, making them difficult for current AI models to evaluate.

While AI's capabilities are rapidly advancing, the nuanced understanding of emotional and experiential quality remains a significant bottleneck. This area continues to be a crucial domain for human testers, who can provide invaluable feedback on user sentiment.

Startup founders in the AI space must critically evaluate whether their offering is merely a feature that foundational models or incumbent companies could eventually absorb. This assessment is crucial for long-term viability and competitive advantage.

The alternative is to build a standalone product that makes sense as its own business, offering unique value and a defensible market position. This involves focusing on high-impact problems that require a comprehensive solution rather than a simple add-on.

Justin Yuan advised, "You have to ask yourself is is this a you know a feature that uh the foundational models or some incumbent will eventually absorb... versus is this a standalone product that makes sense to exist as its own business."