Testing AI Agents Without a Benchmark: How Air Canada's Team Built Its Own Test Sets

Nathaniel introduces himself as a non-evaluator, saying it has been a long time since the last agent he shipped without any evaluation. He says today he wants to tell the story of how he quit this habit.

Weill says he works at Air Canada, a large company with operations, corporate, commercial and loyalty segments, each with its own challenges. His team delivers AI solutions across all of them.

The core problem, he says, is that the team builds agents for tasks so specific that no public benchmark exists to rely on. He adds that the difficulty is not only the lack of a benchmark: the team's data scientists and AI engineers can build agents but are not domain experts, the domain experts sit in different business units and are very busy, and the team has limited access to them. Together these made it very hard to build any evaluation or benchmark.

Before Air Canada, Weill worked at a company building a Q&A assistant for telecommunications engineers. He describes the domain as huge, with about 10,000 documents plus many vendor documents, and says answering a question requires knowing which document to find, which makes the task complex.

In 2024 the team started building RAG systems. As in the Air Canada work, the team had only limited access to experts and could not grade the answers itself.

The team competed with two external teams and one internal team, in segments rather than in parallel. Weill says their solution actually worked, meaning it delivered value and people liked it and used it.

He is candid that he does not know exactly why it worked better than the other teams' solutions. All the teams had the same documents and the same task, but his solution somehow worked, and he says he will show what he and his team did to make it work.

Weill outlines the RAG principle: a question goes to the system, the system finds relevant documents, and then it formulates an answer based on them. He notes this has existed for a long time.

He says that for people who have tried it, if the right page or documents were found, the answer was right about 90 to 100 percent of the time. In his view, the challenge in this task was therefore finding the right documents.

Weill says the team built a very complex pipeline that grew more complicated over time. Whenever they saw a published technique that promised an improvement, they tried it, but it did not improve their solution. He says these techniques worked on paper for the publications but did not really help them.

More importantly, each time they added complexity they found they broke something else, so something that used to work stopped working. He calls this the real problem, and concludes that complexity doesn't always help.

Weill says an agent can be changed in a few minutes, for example by adding a layer of query writing or a reasoning step. Grading that change, however, required a weekly meeting with experts. The team put five different chatbots' answers to the same question on a giant screen and asked the experts to grade them.

Experts were not always in agreement; with several experts in the room, the team got three opinions. Their budget allowed only one solution to be tried per week, which was not enough to go fast.

The process was: ask a question, the agent answers, and an expert reads the answer, sometimes looking things up because they did not know it, which took a lot of time. The team had a budget of one trial per week.

Weill describes a known technique in which the team starts with a document page, asks an LLM to write a question about that page, which he calls an easy task, and then asks the agent to search for the answer and compare what it found to the page used to generate the question. This comparison can be graded in code, with no opinions needed. He says he did not invent the technique but found it very smart.

The impact was that the team went from one evaluation a week to dozens a day. The accuracy of the expert grading kept going up, removing a lot of complexity helped and was not needed, and focusing on the right embedding model, chunking strategy and BM25 alpha value was the right thing to do. Accuracy rose over several weeks.

Weill says he does not know exactly why his team succeeded, but he is fairly sure the other team had no test set and was instead using a long loop with experts, perhaps more expert than his team had access to. His lesson, which has stayed with him for two years, is that if you cannot grade the agent, you cannot make it better.

In 2026 the team worked on analyzing supplier contracts. Air Canada has many suppliers and receives many contracts, and experts check each clause one by one, judging whether it is compliant or needs rewriting. Weill says the task is long and costly, since a delayed supplier delivery causes delays in the chain, and missing a clause can backfire on the company.

The data science team was asked to build an agent that could automate this work. Weill notes that there are many documents and rules to consider, that experts are very busy and expensive, and that there is no benchmark for it. The team could not find an existing benchmark on the web that they could apply, so they faced the same slow-cycle problem as in 2024.

Weill says that in RAG, creating answers from documents is well known and defined, but for this project nobody could say which technique to use. So the team reapplied what it had learned from RAG: getting the answer first and then testing whether the agent could find it back.

He describes the agent in simplified form as having four steps: spotting the clause in the contract, finding relevant documents to support an amendment, writing the amendment, and a classification that says whether the amendment should be kept. Spotting, for example, means classifying all sections of the contract into specific domains, then finding whether there is a defect or a change to make in that section.

The team already had the trick from RAG for finding documents, so that step was covered. For the other steps the team had nothing comparable, so Weill presents five techniques they used or tried.

Weill presents five techniques the team used or tried for this problem, saying his goal is to inspire the audience to reapply them in their own challenges rather than to show off. The first uses past decisions and answers, retrieving past contracts and comparing them with the changed one. The second chooses what you are looking for and writes or generates the contract around it. The third takes a good signed contract and breaks it. The fourth defines things that should not change no matter what, so no extra flags should appear. The fifth, which he calls the most powerful but also very hard, turns one example into many using templates.

The team first tried the approach of asking experts for the original signed contract, taking it, running a diff as in code, comparing what had changed, and checking that the agent produced the same exact flags. Weill says this would be perfect.

However, the team did not keep all of its contracts. It kept the original only sometimes, usually just the last version, and did not have all the versions, since contracts sometimes go into negotiation. Without those pairs of contracts, the approach did not work.

For the second technique, Weill says the team starts from rules rather than from flags. For example, a rule might set a cap above one million dollars that applies to every contract, or to many contracts in specific domains. The team writes a clause that does not break the rule, generates the contract around it, and then checks that the flag, which signals a rule break, is raised. He says this is simple and needs little work, but the problem is that you need to know the rules.

Weill says the team got some rules from the experts or from the documentation, but documentation could not guarantee coverage of everything, so the team also wrote its own rules. In the end it generated 275 tests that it knew could run and whose answers it knew.

He compares this to a RAG test but inverts the problem: the team plants the problem and then tries to find it again.

A third technique, which he calls very hard, takes an example of a signed contract, removes a clause, and expects the agent to say something is missing. Suppliers use different templates with different content, so deciding what should be in which contract takes some thinking. The team mixed these with existing contracts that should show no flag. Weill says this was the most powerful approach they had, and that the agents were missing most of the clauses removed, which he calls a very strong benchmark.

The fourth technique changes something that should not affect the result, such as the supplier's name or address, and checks that the number of flags stays the same. The team also reorders clauses, since that randomness should still lead to the same results, and runs the same contract repeatedly to confirm the output is consistent.

Weill says all of these can be checked using code, with no LLM as a judge. He says the previous speaker showed that LLM-as-a-judge is a very hard task, so the team stays away from it as much as possible.

The fifth technique uses the concept of templates. Weill says the experts have different kinds of contract templates with blanks, which the experts would normally fill in. Instead of trying to fill them manually, the team generates an array of values for each blank.

This produces many contracts: some combinations break rules and some do not, so the team knows which should be flagged as a breach. Again this is a code check with no LLM involved. A benefit is that the experts already use templates, so the team can reuse them to generate contracts.

Weill notes the cycle was slower in this case, about four weeks to get feedback, but after that a script could run and return the evaluation in five minutes.

Weill highlights that exact-match and code-based grading are very fast. Stepping into LLM-as-a-judge can be fancy and useful, he says, but it is very hard. The team got so much noise from the LLM judge, compared with the noise from the agent being built, that solving that problem was almost as hard as solving the original one.

Finding the right prompt and harness for the judge is a very hard task. Padding and fact checking exist, but he says they required so much work that the team could not find the right one for this project.

He briefly mentions other techniques, such as trajectory checks, used in another project where the agent had to complete some hand-off points, and test-to-SQL. His general point is that if you know what you are looking for, you can generate the input and then try to find it back. This does not apply to everything, and sometimes the team had to re-engineer its solution to fit this paradigm, moving slightly away from the original design, because the original design could not be evaluated.

Weill says one thing to highlight is that the work does not end once the tests are generated and a better agent is produced. Experts provide feedback, but the team does not use that feedback to fix the agent directly. It uses the feedback to generate better tests, better prompts and better ideas for how to remove clauses, one loop at a time.

He ends by saying that the tests are not enough on their own to decide whether a product can ship. They are a starting point and a tool to help anticipate developer problems, not the go or no-go decision for the solution. He invites audience members who want to stop releasing agents without evaluation to reach out for support and feedback.