Scaling AI in BFSI: Balancing Trust and Quality

The transition from isolated pilot projects to full production deployment is often stalled by stringent requirements for accountability, auditability, and transparency.

AI systems in BFSI are increasingly probabilistic and non-deterministic, moving beyond simple rule-based automation. This inherent uncertainty makes traditional quality control methods inadequate and raises profound questions about reliability.

Reputational risks in banking and insurance magnify the consequences of AI failures. Mistakes that might be minor in other sectors can lead to severe loss of customer trust and significant financial repercussions within BFSI.

Brian Corkery highlighted that the challenge extends beyond deployment, stating: "the question isn't simply how do we deploy AI it's a much harder question right how do we scale AI while maintaining trust and quality when the systems themselves are increasingly probabilistic, adaptive, and non-deterministic."

Traditional quality control and assurance are transforming into automated, AI-led systems that require stringent oversight.

Domain expertise is becoming indispensable for governing and validating AI outcomes, ensuring that these systems align with specific business contexts and regulatory requirements. Anand Moorthy underscored this shift, noting, "If you really think about it, the governance of AI is the going to be the future because what is right in the domain that we are is going to predict where the future is going to be."

Financial institutions are leveraging generative AI to empower associates and accelerate daily tasks, moving beyond theoretical applications to practical implementation. This involves adopting AI for various functions, from code development to drafting documents.

Brianna Elsass explained the focus on integrating AI into existing workflows: "we are making sure that all of our associates... is really starting to get their hands on AI, figuring out the best places to leverage it in order to accelerate what they're doing and have really started to expand the use cases for our associates". The strategy emphasizes iterative human-in-the-loop workflows rather than fully autonomous agents, aiming to enhance productivity while strictly adhering to organizational standards and compliance.

Instead, fundamental questions around trust, accountability, and explainability frequently halt progress in sectors like insurance and banking.

Eshwar Pastapur articulated these core concerns: "So what stop is it's it's the trust, right? Can you trust the answer? Can I explain it? Can I audit it? What happens when it is wrong? Who is accountable?" These questions highlight the need for robust frameworks that address how AI decisions are made, verified, and attributed.

Quality engineering is emerging as the critical link between a promising proof-of-concept and a stable, auditable production system. It provides the necessary methodologies and tools to answer these questions and build confidence in AI deployments.

The rise of generative AI demands a fundamental re-evaluation of what constitutes 'quality,' moving away from traditional binary pass/fail testing. Eshwar Pastapur noted this shift, stating, "the first thing we have to let go of is the idea that quality means getting exactly the same answer. That's not going to happen, right?"

Generative AI systems can produce multiple, equally valid outputs for the same input, making the expectation of a single correct answer obsolete. This inherent variability necessitates a new approach to validation.

Quality must now be assessed based on whether AI outcomes align with expected objectives and business value, rather than on an exact match to a predetermined response. This perspective allows for the acceptance of diverse yet acceptable results, reflecting the probabilistic nature of advanced AI.

This involves strategically identifying critical moments where human eyes are necessary to verify AI outputs and processes.

Brianna Elsass emphasized this integration: "I think integrating that human in the loop process... I want to know where are those critical moments in that development cycle of the product or the feature that I'm inserting human eyes." Human involvement ensures objective alignment, adherence to style guides, and enforcement of guardrails, which are vital for maintaining compliance and trust. As confidence in AI systems grows, the specific role of human oversight is expected to evolve, becoming more strategic and less granular over time.

Eshwar Pastapur explained, "We accept as a rule there could be multiple outcomes... I think what we'll have to determine is it is a multi-dimensional kind of an stress scorecard of some sort... We have to come up with that that would then test it along the journey of the whole workflow."

This approach allows for a more nuanced evaluation of AI systems, accounting for various valid outputs and their potential impacts. Such scorecards are essential for capturing and reporting the upstream and downstream effects of AI decisions across the entire workflow, providing a comprehensive view of system behavior and reliability.

This shift means that business users may increasingly serve as the first line of defense in testing AI systems. Their intimate understanding of the business context and expected outcomes is crucial for validating AI performance and ensuring relevance.

Industries are transitioning from generative AI to more advanced agentic AI, where multiple AI agents interact autonomously. This evolution necessitates a fundamental shift in testing methodologies, moving from verifying intelligence to validating autonomy.

Eshwar Pastapur described this change: "What you're in a way saying is we going from an AI genai kind of a world into more of an agentic AI multi-agents talking to each other... it goes from testing intelligence to testing autonomy." The focus must now be on testing the entire workflow journey to ensure that autonomous agents make correct decisions and prevent unintended missteps.

This expanded scope of testing is crucial for building reliable and trustworthy AI systems capable of operating with minimal human intervention.

Eshwar Pastapur noted, "I talked about trust automation here even you have autonomy needs to be earned in a way right you start with low risk you start with something small." This strategy involves beginning with low-risk tasks to build confidence and refine processes.

Human-in-the-loop verification remains mandatory in the initial stages, ensuring that AI decisions are reviewed and validated before full automation. This continuous oversight helps mitigate risks and fosters trust.

Monitoring for bias and data drift is essential for the long-term health and reliability of AI systems in production. Autonomy is progressively earned as AI demonstrates consistent performance and adherence to safety and ethical guidelines.

Brianna Elsass suggested, "I would say testing is actually a great place to start to pressure test and scale with pace for enterprises."

AI-driven test automation creates a 'rinse and repeat' efficiency, allowing organizations to iterate quickly and refine their AI models. This accelerates code design and overcomes traditional testing bottlenecks.

Successful implementation of AI in testing builds internal comfort and confidence, paving the way for more risk-prone deployments across other enterprise functions. It provides a controlled environment to validate AI's effectiveness and address potential issues before broader integration.

Traditional programming skills are increasingly augmented by AI intuition, allowing non-technical individuals to take on roles as stewards of AI systems. This lowers the entry barrier into technical roles, broadening the talent pool.

Future engineering requires a stronger emphasis on domain knowledge and orchestration skills over pure structural coding. Engineers will need to understand how to integrate, manage, and optimize AI tools within specific business contexts, rather than merely writing code from scratch.

This shift enables a more diverse workforce to contribute to AI development and deployment, leveraging their understanding of business problems to guide AI solutions.

Eshwar Pastapur stated, "Anything to do with AI, you will have to in a way unlearn what you have had before and then learn it because it's a different muscle to develop right."

This involves developing a different 'muscle' for problem-solving, one that is more adaptive and iterative to accommodate AI's dynamic nature. Companies must prioritize comprehensive retraining programs to align their existing workforce with these new AI-first workflows and mindsets.

Anand Moorthy emphasized this point, stating, "what would differentiate companies in my mind is uh how fast um the velocity in which can they can deploy with confidence these AI tools."

Institutional capacity, encompassing a mindset of agility and a commitment to rapid deployment, will be crucial. This involves aligning talent, developing domain-specific models, and establishing robust testing frameworks.

Agility and velocity are identified as the most critical factors for long-term competitiveness in an increasingly AI-driven landscape. Companies capable of quickly integrating and iterating AI tools will gain a significant market advantage.

Anand Moorthy stated, "I think the human intervention with knowledge of what is the outcome is going to be the key in the days forward. uh because there is no substitution to that piece."

Effective governance requires a human-managed 'AI on top of the AI' to ensure alignment with ethical and business objectives. While AI can provide extensive data and options, it cannot replicate the nuanced analytical thinking and emotional intelligence inherent in human decision-making.

Human intervention serves as the fundamental repository of knowledge needed to validate AI outcomes, especially in complex, ambiguous, or ethically sensitive scenarios. This combination of human insight and AI capability is critical for trustworthy and responsible AI deployment.

Brianna Elsass noted, "I think there is a technology stack overhaul that is going to have to be um evaluated as as well." Organizations must strategically decide where to deploy AI, balancing customer-facing applications with internal operational enhancements.

Legacy systems must be re-evaluated and restructured to work harmoniously with new AI capabilities. This often involves significant architectural changes to ensure data flow, processing power, and interoperability.

These structural transformations are not merely incremental upgrades but fundamental shifts required to build a sustainable and AI-ready technological infrastructure for the future. Without this overhaul, the full potential of AI integration cannot be realized.