Salesforce Shifts to Agentic Testing for AI

Salesforce is currently in the process of evolving its testing strategy for Service Cloud, responding to significant changes within the industry over the past few years. Traditional software testing has historically been deterministic, relying on known inputs and outputs to predict system behavior. However, the rise of generative AI and agentic AI introduces dynamic and non-deterministic elements, fundamentally altering how software behaves and requiring a new approach to quality assurance.

This new era means that outputs can vary significantly from one run to the next, and critical decisions are often made at runtime. The clear pass/fail criteria of old are rarely binary with AI agents, necessitating a more nuanced understanding of quality.

Conventional software allows for straightforward tracing of single-point failures, as its deterministic nature ensures a predictable path from input to output. This predictability enables quality assurance teams to easily identify and isolate issues within a well-defined system.

In contrast, AI agents introduce multiple valid paths and varied outcomes, making it difficult to pinpoint exact failure points. This complexity means that failures in AI agents can directly lead to a loss of customer trust, as unpredictable or incorrect behavior can significantly impact user experience and reliability.

Initially, the primary focus was on ensuring the accuracy, conciseness, and completeness of content generated by their AI systems, addressing fundamental concerns about output quality.

As these AI solutions moved from development into real-world production environments with specific customers, Salesforce began to realize the broader implications of AI behavior. This transition revealed that testing in isolation, or "in a vacuum," did not fully prepare agents for the complexities and unpredictability of live deployment.

The real-world applications highlighted a critical gap between theoretical testing environments and the dynamic nature of production. The insights gained from customer adoption emphasized the need for a more comprehensive testing strategy that accounts for how AI agents interact with actual data and user scenarios.

This shift led to an increased emphasis on understanding the impact of AI in practical use, moving beyond basic content generation checks to a focus on overall system reliability and performance in live settings. Salesforce's approach evolved to bridge the gap between initial accuracy checks and the nuanced demands of deployed AI.

This platform allows customers to incorporate their unique learning data and fine-tune agent behaviors, ensuring greater relevance and effectiveness within their operational contexts.

The studio enables persona simulation, allowing customers to test how agents perform under various user profiles and scenarios. It also provides custom evaluation capabilities, offering tailored metrics to assess the outcomes of AI agents and measure their alignment with business objectives.

Salesforce's new testing approach allows customers to incorporate AI agent evaluations directly into their continuous integration/continuous deployment (CI/CD) pipelines. This integration ensures that agents are consistently assessed throughout the development lifecycle, preventing performance degradations from reaching production.

The system enables detailed tracking of agent traces and provides a comprehensive run history, allowing teams to monitor the various steps an agent takes to complete a task. This capability helps measure the impact of prompt tuning and other adjustments on agent accuracy and overall performance, providing critical insights for continuous improvement.

Deterministic tests continue to hold significant importance within the testing framework, despite the shift towards agentic AI. These legacy tests, which verify predictable behaviors with known inputs and outputs, are not obsolete but rather form a foundational layer for stable software components.

However, it is crucial to augment these traditional tests with agentic evaluations to address the non-deterministic aspects of AI. This involves testing both the final responses and the intermediate process steps an agent takes, and specifically validating against unauthorized information disclosure to maintain data security and privacy.

A robust test architecture must include environments that accurately represent real-world scenarios to ensure the reliability of AI agents. This involves using industry-specific language and grounded data within testing, which is vital for preventing discrepancies between internal development assumptions and actual customer usage.

Such realistic environments help validate that AI agents perform as expected under diverse and complex conditions. This proactive approach helps avoid situations where an agent's behavior differs significantly in production compared to its performance in isolated testing, ultimately improving customer satisfaction and trust.

Manual testing methods are inherently limited in their ability to scale effectively, especially when dealing with the high volume and complexity of AI agent interactions. This bottleneck hinders the comprehensive evaluation needed for modern AI systems.

Leveraging specialized test agents allows for significant scaling, enabling the execution of hundreds and thousands of concurrent calls. This automation is critical for efficiently monitoring the quality of voice agents and other AI systems at scale, ensuring consistent performance across numerous interactions.

It is crucial to integrate AI agent evaluations as a regular task within the CI/CD pipeline, rather than treating them as a final check before customer release. This ensures consistent quality assurance throughout the development lifecycle.

AI agents are prone to fluctuations in accuracy and performance, making continuous testing essential. Regular evaluations guarantee that agents consistently operate at peak performance, preventing unexpected issues from impacting user experience in production.

The unit under test for AI agents is no longer a static piece of software but a dynamic customer journey, requiring a comprehensive evaluation that spans various tools, data, and languages. This holistic view acknowledges the complex interactions and varied outcomes inherent in agentic systems.

Salesforce implements a continuous feedback loop: Evaluate, Observe, and Enrich data. This cycle ensures that agent performance is constantly monitored, insights are gathered from real-world usage, and data is improved to refine agent behavior.

This healthy, iterative process is critical for ensuring the proper and effective release of AI agents to customers. By continuously adapting and improving based on feedback, Salesforce can maintain high quality and customer satisfaction.

The dynamic nature of agentic AI necessitates this cyclical approach, where lessons learned from observation and evaluation feed directly back into data enrichment and agent refinement. This continuous improvement model is essential for navigating the complexities of AI development and deployment.