Quality Engineering Shifts to AI, Risk Management

This approach aims to leverage reliable data for identifying risk areas effectively, rather than to replace the critical role of engineers. The goal involves transitioning current testers into new roles where they can empower other engineers to conduct their own testing, thereby integrating quality more deeply into the development lifecycle.

Karnib's strategic bet is on a fundamental shift in how testing is perceived and executed within organizations. This involves a move away from traditional, labor-intensive manual automation processes.

The emphasis is on creating a robust framework where AI tools augment human capabilities, allowing engineers to focus on higher-value tasks such as risk analysis and system-level validation.

He stressed that organizations need a solid foundation in data and automation before AI can significantly impact quality engineering efforts. Vodafone Turkey is currently rebuilding its automation framework to support both legacy tools, such as Selenium and Playwright, and newer agentic workflows.

This rebuilding effort ensures that the underlying infrastructure can effectively feed AI systems with reliable data, making AI a more potent tool for enhancing testing services and overall product quality.

Akshit Lomash, Director of Engineering at Alchemy, highlighted his focus on democratizing engineering roles this year, specifically blurring the lines between Software Development Engineers in Test (SDETs) and Software Development Engineers (SDEs). The objective is to enable SDETs to contribute directly to feature development, while SDEs are encouraged to take ownership of testing their own code. This approach seeks to reduce friction between teams and enhance overall test coverage by leveraging AI tools more effectively.

This shift aims to foster a collaborative environment where quality is a shared responsibility, leading to more comprehensive testing and higher product quality across the development process.

Panelists unanimously rejected the term 'deprioritization,' asserting that activities are not being discarded but rather strategically shifted. The focus is on replacing manual, repetitive tasks with more efficient automated or agentic solutions. This re-evaluation ensures that critical aspects of quality assurance are maintained and even enhanced through modern techniques.

Manual cycles are not being eliminated entirely but are being replaced by tighter, more frequent automated cycles. This transition helps maintain quality confidence and allows teams to address issues more rapidly.

The strategic shift frees up human capital from mundane tasks, allowing engineers to concentrate on complex problem-solving and higher-value activities that require human judgment and creativity.

By leveraging automation and AI, organizations can achieve a more dynamic and responsive testing environment, ensuring that quality remains a continuous process rather than a sporadic effort.

Kumar Gaurav Jain stated that tool consolidation should not be the ultimate goal in quality engineering, as different teams often require distinct platforms and technologies for specific problems. Instead, the emphasis should be on standardizing processes, particularly reporting and CI/CD pipelines. This approach allows engineering teams the flexibility to choose the most appropriate tools for their unique challenges.

Establishing standardized processes enables the creation of ecosystems that support reusable automation across various domains, including browser, API, and accessibility testing. This ensures consistency in quality while accommodating diverse technological needs.

Such a strategy promotes efficiency and innovation by providing a clear framework for collaboration and data exchange, rather than imposing a uniform set of tools that may not suit every team's requirements.

Tenny Agustin emphasized that tool consolidation is a secondary benefit that naturally follows effective process optimization. He argued that the true burden in quality engineering often lies within inefficient processes, rather than in the number or cost of tools. Prioritizing process refinement allows teams to streamline workflows and identify where tool reduction would genuinely add value.

By optimizing processes, engineering teams are empowered to exercise their creativity and problem-solving skills, leading to more innovative solutions. Strong leadership is essential to guide this optimization and prevent chaotic tool sprawl, ensuring that any consolidation efforts are purposeful and yield tangible benefits.

To prevent quality engineering from devolving into a 'Wild West' of disparate tools, strong leadership and clear governance are essential. Tenny Agustin explained that a robust framework of standards must be in place, collaborating with other engineering leaders to ensure desired outcomes are met. This structured approach helps manage the proliferation of tools and ensures each serves a strategic purpose.

Proof of Concepts (PoCs) and discovery phases are crucial for evaluating new tools, allowing teams to assess their efficacy and alignment with business goals before significant investment. This due diligence ensures that only tools delivering clear business outcomes are integrated into the workflow.

Avoiding over-investment in unvetted tools is a critical aspect of this governance, preventing wasted resources and maintaining focus on solutions that genuinely enhance quality and efficiency.

The primary metrics now focus on defect escape rates and incident rates, which directly reflect the quality experienced by end-users. This change directs executive attention to actual production outcomes rather than internal testing progress.

This new focus ensures that the effectiveness of testing efforts is judged by its impact on the final product and customer satisfaction. High incident rates, despite a large number of tests, indicate a fundamental failure in the quality process.

The ultimate goal is to build confidence in the platform by ensuring a seamless and reliable experience for customers. This perspective reorients quality efforts around the user's perception and satisfaction.

Prioritizing backend strength is crucial to minimize the discovery of edge cases by users, ensuring that potential issues are addressed before they impact customer interaction. This proactive approach reinforces trust in the platform and reduces the likelihood of negative user experiences.

While AI is currently utilized for automation report analysis and evaluation, the goal is to streamline workflows without compromising quality. She noted that shortening timelines has proven difficult despite AI integration, indicating deeper systemic issues.

Saini explained that achieving tangible results with AI in quality engineering requires a fundamental shift in how teams operate. This includes adapting existing processes to integrate AI effectively, which often means an overhaul of traditional methodologies.

The expectation of immediate, drastic improvements with AI needs to be tempered with the understanding that successful adoption involves a long-term commitment to evolving organizational culture and individual mindsets. Without this, AI tools may not yield their full potential.

Core considerations still include automation results, defect analysis, and manual validation. However, the reliability and precision of AI-generated outcomes now play a significant role in determining release readiness.

The introduction of AI necessitates a more rigorous evaluation of its outputs to ensure they align with expected quality standards. This means that teams must not only rely on AI for efficiency but also verify its accuracy to prevent the introduction of new risks.

Maintaining zero defect levels is paramount to preserving customer trust in the platform, especially as AI becomes more integrated into the testing and release processes. The AI's contribution to this goal directly impacts its credibility.

Ali Karnib categorized AI initiatives in quality assurance into three distinct buckets: those that have paid off, those that show promise, and those that have fizzled out. Successful applications include automating monotonous tasks like test case writing and script maintenance, significantly boosting efficiency.

Initiatives like consolidating test cases have shown mixed results, often encountering challenges due to missing context and high token costs associated with AI processing. This indicates that while AI can streamline some consolidation, it struggles with the nuanced understanding required for comprehensive merging.

Promising areas involve leveraging AI for coverage expansion, particularly across vast and complex configurations, such as over 400,000 combinations. This application demonstrates AI's potential to enhance the breadth of testing, identifying issues in areas difficult for human testers to cover comprehensively.

Kumar Gaurav Jain stated that AI has not yet reached a point where it can fully replace human judgment, especially in complex decision-making scenarios. While AI excels at enhancing engineering productivity, managing test data, and optimizing execution, its role is primarily one of augmentation rather than outright replacement of engineers. He remains in a 'wait and watch' mode for several AI applications.

AI's strength lies in its ability to process vast amounts of data and automate repetitive tasks, but the nuance and intuition of human judgment are still indispensable. This distinction highlights that human engineers will continue to play a critical role in overseeing and guiding AI-driven processes.

Ali Karnib emphasized that relying solely on AI for such critical decisions introduces a significant risk, underscoring the human's role as a validator and arbiter. This ensures that complex problems benefit from nuanced human understanding and experience.

The human element is crucial for navigating ambiguous situations where AI might lack the contextual understanding or ethical framework necessary to make optimal choices, thus safeguarding the integrity of the decision-making process.

Leaders must prioritize maintaining trust in the system, ensuring that increased speed from AI does not compromise reliability or quality. This ongoing human verification guarantees the integrity of AI-driven testing processes.

Human intelligence serves as a critical checkpoint, providing the nuanced understanding and contextual awareness that AI systems currently lack. This oversight ensures that any discrepancies or errors introduced by AI are identified and rectified promptly, preserving the overall quality of the product.

The role of AI is to assist and augment human capabilities, not to replace them entirely, particularly in areas where critical judgment and reliability are paramount. This collaborative approach leverages the strengths of both AI and human intelligence to achieve superior outcomes.

Ali Karnib predicted that writing tests will no longer be a primary task for SDETs. Instead, their role will involve reviewing agent-generated tests and accepting which risks are valid, shifting the focus from test creation to risk assessment.

Leaders are now tasked with teaching their teams judgment skills over technical syntax, emphasizing decision-making capabilities as crucial for the future of quality engineering. This transformation requires a re-evaluation of current skill sets and a focus on analytical thinking.

Hiring criteria are also adapting, prioritizing candidates who demonstrate strong decision-making abilities and risk management acumen over those solely proficient in automation metrics. This reflects the industry's pivot towards a more strategic approach to quality.