DeepEval: Evaluating LLM Relevancy

Traditional software testing relies on deterministic assertions such as string or JSON matching to verify expected outcomes. This approach is effective for predictable software components, ensuring exact matches for predefined conditions. However, Large Language Models (LLMs) break this traditional model due to their non-deterministic nature and susceptibility to hallucinations, rendering exact string matching ineffective for quality verification.

Monika Sharma highlighted this shift, stating, "Traditionally u we how we used to test is we usually rely on deterministic assertions like uh for example string assertions or JSON assertions... but LLM breaks this model many teams report hallucinations issues outputs are nondeterministic in nature."

This framework scores outputs on a scale from 0 to 1 based on custom relevancy checks, providing a nuanced assessment of quality.

Monika Sharma explained, "Deep is an open-source evaluation framework known as pi test for llms. So instead of comparing string, JSON images uh it uses judge model to evaluate output semantically."

DeepEval's technical workflow begins with capturing input and output pairs from an LLM application, packaging them as 'LLM test case' objects. These objects, which contain both the input query and the generated response, are then fed into a metrics engine. The engine utilizes an LLM judge to score the responses based on predefined criteria, ultimately generating a pass or fail verdict along with the reasons for the outcome.

Monika Sharma described the process: "The flow here is your LLM application... produces output. So these output are captured as test cases which are LLM test case object containing input and output. These flow into metrics engine which uses LLM judge to score them."

RAG evaluation focuses on critical aspects such as faithfulness (accuracy of the response to the retrieved context), answer relevancy, and the quality of the context itself, assessed through precision and recall measures.

Agent evaluation is designed to track how effectively an LLM agent completes its assigned tasks and the correctness of the tools it employs. Safety metrics address potential risks by covering issues like toxicity, bias, Personally Identifiable Information (PII) leakage, and adherence to role-based guidelines. Monika Sharma stated, "Deep ewell offers metrics organized into different categories. The categories are listed here which is rag evaluation, agent evaluation, safety compliance, custom and conversational."

Essential dependencies required for the evaluation setup include pandas, numpy, and pytest, foundational libraries for data manipulation and testing within Python.

Additionally, the Google Generative AI SDK is specified for evaluating API-based LLM models. Monika Sharma explained, "the different thing which I am doing here is using uv init to initialize python project... just like we have npm in it for node based project likewise UV in it is Python based one."

Given that Large Language Models (LLMs) produce varying responses each time, setting a fixed threshold for evaluation is crucial. A threshold of 0.8 is established to account for these inherent variations in LLM outputs, allowing for a robust assessment of quality. The evaluation encompasses multiple metrics, including relevancy, correctness, and completeness, to provide a comprehensive view of the LLM's performance.

Monika Sharma noted, "So in that way we are keeping a threshold as 0.8."

When evaluation results display 'None' instead of a numeric score, it signifies an output format mismatch from the LLM agent. This typically occurs when the agent's response deviates from the expected structure DeepEval is configured to process. Such discrepancies can arise from issues like excessive response length or the inclusion of unexpected extra fields in the output, preventing proper evaluation.

Monika Sharma clarified, "that's because the output by agent was not in a format where deep evil has used for other test."

This functionality allows for the definition of specific evaluation logic beyond the default relevancy checks provided by the framework. Custom metrics are particularly useful for tailored assessments, including safety compliance testing to identify issues like toxicity, bias, and Personally Identifiable Information (PII) leakage.

Monika Sharma explained, "for example the way we have relevancy likewise we can create our own custom metrics via GEL."

Just like any other software tests, if LLM evaluations are not incorporated into the pipeline, they fail to provide actionable data and insights. This integration ensures that models are continuously monitored and validated, catching regressions early in the development cycle.

DeepEval tests are compatible with popular CI/CD platforms such as GitHub Actions, GitLab CI, and Azure Pipelines. Monika Sharma emphasized, "reason why we are we have to put in CI/CD because just like any other test if we don't put it in our pipeline it's not going to give us any feedback."

If the objective is to check direct API calls and the immediate response of an agent, the evaluation acts as a unit test. However, if the aim is to verify the behavior of a chatbot within a user interface, considering the entire system's interaction, it functions as an integration test.

Monika Sharma clarified, "in case if our target is just to see agent response... it properly fits in unit test but if our objective as a QA... it fits into integration test."

DeepEval demonstrates strong efficacy in evaluating text-based LLM responses. However, its performance significantly diminishes when user interfaces incorporate non-text components such as images, emoticons, or other visual UI elements. In such multimodal feedback scenarios, DeepEval struggles to recognize and properly assess these diverse components, indicating a current limitation in its capabilities.

Consequently, human judgment remains superior for evaluating interactions that involve a rich mix of textual and visual information. Monika Sharma admitted, "DeepEval is not able to recognize that properly in case of different components except text. So that is where I I think that human judgment won but yeah I mean um in case there are multiple components involved except text then I think DeepEval might not be a great fit."

It is critical to test individual utterances and interactions of each sub-agent within the system. This approach ensures comprehensive verification of tasks handled by different sub-agents, such as those involved in onboarding processes or billing inquiries.

Testing must encompass a diverse range of utterances, including multilingual inputs and special symbols, to accurately assess an agent's robustness and capability. Monika Sharma emphasized, "So in that case it can't be final outcome. Okay. So um in those for those different sub-agents there are multiple utterances which we can which we have to test with it just not one utterance... we have to go ahead with different types of utterances."

A strict prohibition is enforced against AI agents having write access to their own training or knowledge data. This restriction is primarily driven by security concerns, as allowing agents to modify their knowledge base could introduce vulnerabilities or unintended alterations. Consequently, every organization implements constraints on knowledge base updates, limiting agents to read-only access for Retrieval-Augmented Generation (RAG) purposes.

Monika Sharma unequivocally stated, "Strict no. Because of obvious reasons because of security reasons it should not be allowed and every organization has put a constraint on knowledge base updations. It just read access which is provided to agent to use with RAG but I mean but answer to this is no."

Initial testing typically begins with a 0.5 threshold to assess the agent's response quality. As the agent demonstrates stability and a reduced rate of hallucinations, the threshold can be gradually increased, potentially to 0.8. The adjustment of thresholds is directly dependent on the observed stability of the model.

Monika Sharma explained, "There is not exactly one number. So while testing how it goes is that we always start with 0.5 to check for the quality of agent's response, and then as we see that our agents are stable they are not hallucinating much... then we go ahead with maybe 0.8 basically but depends on model stability."

Achieving production autonomy for LLM agents necessitates rigorous testing across multiple, diverse test runs. This involves subjecting an agent to large datasets, such as 40 or more varied utterances, to thoroughly evaluate its relevancy, precision, and completeness. Confidence in an agent's readiness for production is built through consistent performance observed over numerous automated runs.

Without stable and predictable results from these comprehensive tests, there is insufficient evidence to justify deploying the agent into a production environment. Monika Sharma stressed, "We have tested with multiple utterance maybe 40 different utterances hitting same agent and then checking relevancy precision completeness... what gives us confidence is the runs the automation runs... if the results are not stable, it won't give us confidence and there is no proper reasoning that it should go in production."