Benchmarking AI: A Five-Step Framework

Publicly published AI benchmarks are increasingly unreliable because many are now included in the training datasets for new models, according to Nathaniel. This integration means models are trained on the very metrics intended to objectively evaluate them, diminishing their value.

Frontier AI models are being released at an average rate of one every 11 days, making it difficult for benchmarks to remain current and unbiased.

While companies are not intentionally misrepresenting results, the continuous inclusion of benchmarks in training data compromises their ability to serve as objective measures of performance over time.

Selecting representative tasks, identifying candidates, choosing an evaluation method, anonymizing outputs, and measuring the performance-to-cost ratio form the core of the five-step evaluation framework Nufar Gaspar created. This framework aims to make the evaluation more systematic and applicable to individual work.

The process involves selecting diverse tasks that are representative of actual work, identifying potential model candidates, choosing an evaluation method (manual, scripting, or dedicated platforms), anonymizing results to prevent bias, and finally assessing if the performance improvements justify the cost and effort of switching tools.

To ensure effective and unbiased AI model evaluation, it is advisable to hide the names of the models or tools that generated each result. Users often exhibit a preference for tools they are already familiar with, leading to subjective assessments.

Nufar Gaspar highlighted the importance of this approach, noting that even she was surprised by her own test results when model names were hidden, underscoring how deeply ingrained biases can affect judgment.

New models frequently enable previously impossible use cases, reinforcing the importance of manual testing even when workplace toolsets are strictly limited.

One common objection is that the AI frontier is moving too quickly for manual testing to be relevant, but new models frequently unlock previously impossible functionalities, making current assessments valuable.

Another concern is workplace restrictions on tool usage, but even with limited options, comparing different configurations or using mock data can build a business case for new licenses.

Nufar Gaspar suggests that understanding what constitutes 'bad' output can sometimes be more informative than focusing solely on what 'good' looks like, particularly when evaluating nuanced use cases.

When conducting a blind AI evaluation, it is recommended to test an existing model against one to three new candidates, limiting the total to four to avoid review fatigue, according to Nufar Gaspar. Including the current tool establishes a baseline to measure the true benefits of a potential switch.

To prevent bias or context interference, each request should be run in a fresh chat environment. Throughout the review process, model names must be hidden to ensure objective scoring, allowing evaluators to focus purely on output quality.

Testers can utilize friends, spreadsheets, custom scripts, or API-based shuffling to mask the source of outputs and eliminate brand bias. There are several methods to achieve this.

These methods include having a friend anonymize the results, using a spreadsheet to generate random numbers for shuffling, leveraging an AI tool to re-label outputs, or employing a custom script. Automated approaches typically require API keys from services like OpenRouter or individual vendors.

Absolute scoring on a scale of 1 to 5 is often challenging because humans lack consistent internal standards for evaluation. Nufar Gaspar explained that it is generally easier to compare several alternatives for a specific task side-by-side.

This comparative approach allows users to identify specific 'X-factors' that drive their preferences, which are often hard to articulate when scoring in isolation.

Ultimately, the evaluator's own judgment and taste remain the most critical metrics for personal AI benchmarking, as these subjective elements are best captured through direct comparison.

AI models can produce varied responses even when given identical prompts and configurations, highlighting their stochastic nature. For critical tasks with financial or other significant implications, running the same request multiple times per model is essential.

This repetitive testing allows for a more comprehensive comparison of the diverse set of responses. Consistent execution helps validate findings and ensures the benchmark's reliability, accounting for the inherent variability of AI outputs.

A blind testing methodology ensures unbiased evaluation by using an API to generate responses from multiple AI models, according to Nufar Gaspar. The results are then anonymized, typically labeled A through G, so users cannot identify the origin of each output.

Evaluators score the models based purely on personal preference and provide verbal feedback for each result.

After the scoring phase, a 'reveal' process links the anonymized scores back to the specific model names.

This method ensures that personal biases towards particular brands or tools do not influence the assessment of output quality.

To ensure benchmarks reflect real-world workflows, Nufar Gaspar suggests manually refining the request and the scoring guide instead of relying on generic task descriptions.

A benchmark is only effective if it accurately represents the user's real-world workflow and helps them make informed decisions about specific usage scenarios.

A 1-5 numerical rating paired with a verbal evaluation determines how a model performs, as the presenter critiques common AI-typical patterns like the 'A and not B' structure. Nufar Gaspar, for instance, critiqued a model for using typical AI writing patterns, specifically an 'A and not B' structure she personally disliked.

In her evaluation, none of the models produced content that was ready for immediate posting without further editing, indicating the current limitations of AI in generating fully polished content.

The evaluation process allows users to switch between their personal scores and those provided by an AI judge. When activated, the AI judge processes all results and gradually reveals its assessments for each task, as explained by Nufar Gaspar.

The speaker noted significant disagreements between her personal preferences and the AI judge's scores, highlighting the different perspectives and criteria applied by human and artificial intelligence in evaluating outputs.

Because the judge sees only the code and text rather than rendered visual components, Nufar Gaspar finds manual oversight more reliable for visual tasks. For example, a judge's choice regarding a website might be based solely on its underlying code, not its visual presentation.

The speaker expressed a preference for manual evaluation when visual components are critical to ensure accuracy.

While fixing judge instructions is an option to improve AI evaluation, manual oversight remains a more reliable approach for tasks that depend on visual assessment.

The emergence of 'ultra-fast' models has shifted Nufar Gaspar’s priorities, as she noted, "I realized that I'm paying way more attention to speed versus how I did before I've seen the ultra fast demo." Users are now paying much more attention to response times than they did before experiencing these high-speed demos.

These rapid models are reshaping user expectations and priorities for daily AI workflows.

Consequently, evaluation criteria are dynamically updating to reflect the growing necessity for fast responses in practical applications.

Operational tasks are profoundly influenced by the internal system setup and contextual dependencies, making their evaluation more complex. Nufar Gaspar advises exercising caution when judging these tasks without considering their full system context.

She suggests either running operational tasks directly through the actual tool or creating a simplified proxy for testing. Gaspar also warns against using third-party API keys, which may only provide a partial view of a model's true performance within a given operational environment.

Generic instructions such as 'create a good blog post' are insufficient for effective testing of agentic AI tools, according to Nufar Gaspar. Effective prompts must define measurable constraints and specific goals.

Prompts that include explicit requirements, such as targeting specific audiences (e.g., executives and ICs), limiting word count, or specifying required sections, lead to better results.

Highly measurable and attainable prompts ensure more consistent model evaluation and improve the likelihood of achieving desired outcomes from agentic AI systems.