BM25's Unexpected Resurgence in AI Agent Search

Jo Kristian Bergum, CEO of Hornet.dev, initiated his talk by introducing his extensive experience in search and retrieval, spanning over two decades. He outlined the core premise of his presentation, focusing on the unexpected and significant effectiveness of BM25 for agentic search.

He emphasized the intriguing paradox of a 30-year-old lexical scoring function maintaining such high relevance and performance in the rapidly evolving landscape of artificial intelligence.

Unlike traditional one-off queries, agentic search functions as a continuous and iterative process.

This continuous search mechanism enables autonomous agents to systematically gather, process, and refine information as they work towards accomplishing complex objectives.

The model must be proficient in formulating queries and utilizing various tools to interact with information.

A 'harness' serves to expose retrieval functionalities to the model, typically through mechanisms like tool calling or a dedicated code mode, ensuring seamless integration. Finally, a high-performance retrieval engine is crucial for scaling operations across vast quantities of documents, potentially numbering in the millions or billions, to provide quick and accurate results.

This algorithm is fundamentally a scoring function that computes the relevance between terms in a query and terms within documents.

Decades of subsequent research have been dedicated to optimizing and accelerating its performance, particularly in the realm of Top-K retrieval, where the goal is to efficiently find the K most relevant documents.

While the core BM25 scoring function has remained unchanged for decades, its resurgence is attributed to the emergence of powerful Large Language Models (LLMs) acting as highly sophisticated 'users.' LLMs bring a wealth of general knowledge, enabling them to formulate more effective and precise search queries.

These models leverage their implicit understanding of entities, dates, and company names to enhance search performance significantly, making BM25 more effective than it was with less capable users.

BrowseCom+ is a specialized benchmark designed for deep research, comprising approximately 830 riddle-like questions that test an agent's ability to retrieve specific information. It operates on a vast corpus of 105,000 web documents.

This benchmark represents a significant departure from traditional information retrieval evaluation, which typically focuses on human users reviewing a single set of ten blue links. Instead, agentic evaluation acknowledges that agents can generate multiple, iterative queries to achieve their goals.

Modern models typically begin to experience degradation in quality when processing around 350,000 tokens.

This inherent limitation underscores the critical importance of retrieval systems, which are essential for feeding only the most relevant information into the model's context window, thereby preventing information overload and maintaining performance.

Studies reveal that GPT-4 achieves high accuracy when necessary evidence documents are directly provided within its context window, indicating that the model's reasoning capabilities are not the primary bottleneck. However, accuracy significantly declines when the model is required to utilize a retrieval tool to find that evidence independently.

The overall end-to-end accuracy, which measures the model's ability to answer a question using a search tool, heavily relies on two factors: the model's proficiency in formulating effective queries and the inherent quality of the retrieval system itself.

This suggests that enhancing retrieval mechanisms and query generation by LLMs is paramount for improving agentic search performance, rather than solely focusing on reasoning improvements.

Historically, human users have typically employed very short and simple queries when interacting with search engines. In contrast, advanced models like GPT-5 demonstrate behavior akin to 'power users,' leveraging their general knowledge to construct lengthy and complex queries.

These LLMs can effectively utilize sophisticated syntax operators, such as 'site:' and specific phrases, which they have learned from vast amounts of web search data. Agents engage in iterative search trajectories, progressively refining their queries to either fill their context windows with relevant information or to pinpoint precise answers for their tasks.

Recent research challenges earlier conclusions about BM25's performance, indicating that initial benchmarks often utilized inadequate parameter configurations, particularly for handling longer documents. This led to an underestimation of its true effectiveness.

It is crucial to properly tune BM25's hyperparameters to account for the varying characteristics of document lengths and content. Achieving significant accuracy gains often hinges on this careful configuration, implying that a poorly configured BM25 is not representative of its potential.

BM25 offers a distinct advantage for exact matching of specific entities, such as names, zip codes, and SKUs, which can be challenging for embedding models that encode all tokens into a fixed vocabulary. This exactness helps avoid the limitations of semantic representations when precise data is required.

Furthermore, BM25 is computationally more efficient to operate compared to running large embedding models, leading to potential cost savings. Its interpretability also allows agents to better understand and refine their own search queries, contributing to more effective and transparent workflows.

This method addresses the challenge of retrieval across billions of documents by treating them as a Search Engine Result Page (SERP) for agents.

The system organizes these documents into a dynamic workspace, analogous to a traditional file system. This allows agents to interact with a large corpus more effectively, navigating and retrieving information as needed without being overwhelmed by sheer volume.

This allows the model to intelligently decide if more extensive reading of a particular document is necessary.

This strategy combines sandbox environments, retrieval tools, and Virtual File System (VFS) capabilities, leveraging existing primitives found in coding agents to streamline information access and processing. By minimizing the amount of irrelevant information presented, agents can operate more efficiently.

The current optimization strategies for agentic search are seen as a tactical 'hack' designed to leverage the existing strengths of Large Language Models (LLMs), particularly in areas like coding, bash scripting, and tool utilization. This approach recognizes that LLMs are continually evolving, with future model updates expected to enhance end-to-end task performance significantly.

By combining robust retrieval infrastructure with execution sandboxes, the current setup serves as a powerful interim solution, positioning the system to readily integrate with future advancements, ultimately aiming for more sophisticated Artificial General Intelligence (AGI) capabilities.

These classical metrics fail to capture the dynamic nature of agentic workflows.

Agents do not merely consume a single set of search results; instead, they reformulate queries iteratively and perform complex expansions based on previous findings. Therefore, success for agents should be measured by their ability to complete tasks accurately and provide correct answers, rather than by static relevance scores designed for human consumption.

Hornet has invested in BM25 as a foundational primitive for agentic search, resulting in a highly optimized and efficient implementation. Benchmarking tests involving 100 million web documents on a single node demonstrated Hornet's superior performance compared to other generic BM25 engines.

Hornet exhibited significantly higher throughput and reduced latency, underscoring the efficiency of its implementation. This enhanced performance translates directly into substantial infrastructure cost savings for users, making it a highly competitive solution in the market.

A key takeaway is that agents' inherent general knowledge significantly enhances the effectiveness of tools like BM25 and grep in retrieval tasks. It's crucial to recognize that the implementation, performance, and optimal parameters of BM25 can vary widely across different systems.

The explainability of BM25 makes it a strong choice for models, as agents can better understand why certain results are retrieved. Furthermore, combining literal matching capabilities of BM25 with the pattern-matching power of grep creates a highly potent paradigm for agentic search, allowing for both precise and flexible information retrieval.