AI Shifts Testing from QA to Trust Engineering

The conference aims to unite industry leaders to address the challenges and opportunities of scaling quality in the AI era. This year's event highlights the critical shift towards integrating AI while maintaining trust and robust quality standards.

Kishor Wikhe brings over 33 years of experience in the BFSI (Banking, Financial Services, and Insurance) sector and enterprise technology. His extensive background includes leading global delivery center transformations at Capgemini, positioning him as a prominent voice in the evolving tech landscape.

Kishor Wikhe addressed a prevalent misconception among AI enthusiasts: that AI, particularly large language models (LLMs) writing code and automated pipelines, renders traditional testing obsolete. He acknowledged the narrative suggesting "the testing is more or less is the past of the thing." However, he challenged this perspective by posing critical questions about the real-world reliability and safety of fully automated AI-driven systems, emphasizing that the human element of assurance remains vital.

Kishor Wikhe presented a thought experiment to challenge the notion of complete trust in automated AI pipelines, asking if one could "sleep peacefully in the night um you know, knowing the systems won't fail maybe because of the sudden million million user spike that you may see happening, could be data leaks, or even probably I would say generating a hallucinated um you know, response, right? To to a customer." He highlighted potential failure points such as sudden spikes in user traffic, data leaks, and AI hallucinations. When confronted with these critical scenarios, a colleague admitted they lacked complete trust, underscoring the inherent risks that even advanced automation cannot fully mitigate.

AI hallucinations, once viewed primarily as model quality issues, now pose significant real-world liabilities for enterprises. Kishor Wikhe stated, "what it means is the hallucination is no longer a model quality issue. I mean, if you apply the generative AI business uh or a legal filing or your board decision, it really becomes a governance issue." A fabricated reference in a government report, for example, can lead to severe consequences, including loss of credibility, public apologies, and brand damage. The implementation of AI in sensitive sectors, such as legal filings or board-level decision-making, elevates these issues to a critical governance concern.

AI agents, when granted elevated authority, can inadvertently cause significant damage while merely performing their assigned tasks. Kishor Wikhe explained, "The agent absolutely was not malicious or was not trying to create any damage. Was just was trying to pursue the task which he's been asked to do it. And and you know, with the authority which it had uh been given by us. And the real risk is not intelligence, but not the intent, by the way, huh? So, what it means is agent failures are off often, you know, authority failures." He emphasized that the risks stem from 'authority failures' rather than malicious intent, as unintended consequences can arise when agents access sensitive resources with excessive credentials.

Regulatory bodies are increasingly adopting a mindset akin to aviation incident investigations, shifting AI governance from predictive assessments to evidence-based oversight. Kishor Wikhe stated, "The question is more about can you prove what happened after the launch? I mean, that's how I would say the regulators are the future of AI governance is not about prediction. It is about evidence." This shift emphasizes continuous documentation and post-deployment monitoring. Organizations must now demonstrate and prove the actual behavior of AI systems after launch, rather than simply relying on initial trust or model predictions.

The future of testing extends beyond merely finding bugs to an emerging discipline called "Trust Engineering," according to Kishor Wikhe. He explained, "It's basically going to be about the trust uh which has to be engineered across probably the data, the authority, or the identity, escalation monitoring, and also the evidences you know, and that is where the future of testing is bigger than just testing and doing the check box uh and and reporting the reports bugs actually, and the reporting the bugs. It is more about trust engineering." This holistic approach recognizes that model quality alone is insufficient for reliable AI systems, necessitating engineered trust across various dimensions, including data integrity, identity management, and continuous monitoring.

Kishor Wikhe noted, "The question itself is changing. It is basically changing moving from which is a decade old uh you know, the scenario of uh does this feature work as expected as per specification to more of in the today's world, which is actually not enough by the way. Can our users can our businesses or our regulators trust this platform?" Isolated testing can no longer guarantee outcomes at scale in AI-driven environments. The primary question for Quality Assurance (QA) teams is changing from simply "does it work?" to the more profound "can it be trusted?" to reflect this new reality.

And it is you know, basically you know, AI enabled systems just do not depend on codes. They depend on data, models, prompts, context, the human interactions and the failure modes are really really different today." Unlike traditional software, AI failure modes are distinct, meaning systems can pass functional tests yet still exhibit issues like hallucination or model drift. This implies that even if automated pipelines appear operational, business outcomes may remain unstable due to these emergent complexities.

Kishor Wikhe articulated, "I think we already talked about basically that the era of testing as a gatekeeper is kind of, you know, transition to more of the digital transformation area as actually as expanded scope." While traditional testing focused primarily on release readiness, the expanded scope now includes integrating AI systems and assuring the promises made to users, reflecting a move beyond digital transformation to AI-integrated paradigms.

Kishor Wikhe contrasted the two approaches, stating, "In the old model, testing produced a report. In the new model, trust engineering producing the evidence." He emphasized that assurance must be a continuous process, rather than a one-time event, especially as AI models constantly evolve and interact with dynamic real-world environments. This new paradigm demands ongoing validation and proof of trustworthiness.

It is crucial to differentiate between using AI as a tool for testing and testing AI systems themselves, as Kishor Wikhe highlighted. He explained, "Here the testing with AI is nothing but a productivity play, right? You test the testings, leverage that AI as part of your testing, provide the productivity gain. And then testing AI systems is basically you're you're building the trust, you're enabling the trust in the system." Testing with AI primarily focuses on productivity gains and automating existing tasks, whereas testing AI systems focuses on building and enabling trust and reliability within the AI-driven system itself. The latter represents the current and most significant evolution in the quality assurance field.

Testing is not disappearing but rather undergoing a profound metamorphosis, evolving into a strategic function now known as Trust Engineering. Kishor Wikhe affirmed, "I would say the testing is metamorphosizing. It's basically it's transforming. And it is moving from I would say the quality assurance to something much much bigger and broader and far more strategic. And it's really becoming a trust engineering." This transformation requires addressing core questions about whether an AI system is reliable, secure, resilient, and fundamentally trustworthy. The new imperative is to provide empirical evidence that substantiates and justifies this trustworthiness across all aspects of AI integration and operation.

The host stated, "I was joking the other day with with someone that now is the time for chief quality officers because agents and AI is going to be embedded everywhere." With AI and autonomous agents becoming embedded into every system and process, quality engineering will need to extend its reach to every business function, making a dedicated C-suite role essential for overseeing comprehensive trust and assurance.