Exa's Will Bryk on Search Engines for the Agentic Web

The projected growth indicates that AI search activity will eventually be 1,000 times greater than current human levels, fundamentally transforming the search ecosystem.

This dramatic increase signifies a move towards an agent-centric web, where AI agents will be the primary consumers of search results. Will Bryk emphasized the profound implications of this trend, stating that it represents one of the most critical takeaways from his presentation.

Exa currently serves over 5,000 companies and supports more than 400,000 developers, demonstrating its broad impact across various sectors. The platform is utilized by a diverse range of AI agents, from coding tools to financial applications.

For instance, coding agents like Cursor rely on Exa for fetching technical documentation, while go-to-market (GTM) agents such as HubSpot leverage it for generating high-quality leads. Additionally, Exa is widely employed in the financial sector for precise financial data retrieval, showcasing its versatility and critical role in different industries.

With trillions of pages on the web, human synthesis of information has become an impossible task.

He argues that Google functions more like a 'recommendation engine' rather than a true database capable of providing exact answers. For example, a search for 'shirts without stripes' often yields recommended results instead of specifically filtered information, highlighting the limitations in obtaining precise data.

This shift away from accurate, database-like results poses significant challenges for users seeking specific information.

Users often avoid typing such detailed requests because they anticipate a failure to deliver precise results, instead receiving loosely related links or documents.

For instance, a query like 'find everyone in Singapore who works on AI search' typically provides documents containing some of the keywords rather than a curated database of matching individuals. This limitation means users lack confidence in traditional engines to provide comprehensive and accurate data for critical tasks, necessitating a more sophisticated approach to information retrieval.

A robust information infrastructure is crucial for maintaining societal stability and preventing a future dystopia, according to Will Bryk. He warns that failing to address current information retrieval problems could lead to a world plagued by misinformation.

Bryk believes that a lack of accurate information can foster manipulation and result in poor decision-making at both individual and collective levels. He suggests that unchecked AI-driven information environments could exacerbate these issues, creating a future where the absence of truth undermines societal coherence.

Therefore, fixing the information problem is presented as a fundamental step in safeguarding future societal well-being.

The advent of transformer models, particularly GPT-3, has opened new possibilities for superior search capabilities by demonstrating a remarkable ability to understand text input. This advancement led to the concept of combining the power of GPT-3 with search engines to achieve near-perfect information retrieval across the world's data.

The core idea involves using neural networks to precisely match queries to documents, moving beyond traditional keyword methods. However, scaling this approach to process a trillion documents presents a massive technical challenge that requires innovative architectural solutions.

Traditional search engines rely on keywords, which are efficient for simple queries but prove inadequate for complex information requests. To handle more nuanced and sophisticated searches, the shift towards neural networks is essential.

Neural embeddings encode the semantic meaning of documents, allowing for a more intelligent and contextual understanding of information. By combining the strengths of neural embeddings with keyword-based methods, search engines can effectively address highly complex and subtle queries, providing more accurate and relevant results than ever before.

The team, although new to the search domain, was deeply committed to solving the inherent limitations of existing search technologies.

During this period, Exa raised millions in funding and made significant investments in a GPU cluster to support its ambitious R&D efforts. Their primary focus was on addressing search problems from first principles, aiming to create a fundamentally new and more effective information retrieval system.

Exa's initial goal was to build a superior search engine that could outperform Google, without a preconceived notion of an API-first business model. The team was focused purely on technological innovation, intending to figure out monetization later.

However, early users unexpectedly requested API access, signaling a demand for programmatic information retrieval. This user-driven insight prompted a pivotal transition from a consumer-facing search interface to a developer-focused API, allowing other applications and AI agents to integrate Exa's powerful search capabilities directly.

This shift transformed Exa's strategic direction, highlighting the unforeseen needs of the emerging agentic web and establishing a new path for the company's growth.

The demand for API access indicated a broader market need for direct, machine-readable information, which traditional search engines were not adequately addressing. By embracing an API-first approach, Exa positioned itself as a critical infrastructure provider for the burgeoning AI ecosystem, empowering developers to build sophisticated applications that leverage advanced search functionalities.

This marketplace bridges public web data with private, proprietary data sources, significantly expanding the scope of information available to AI systems.

Data providers can set the value of their content, receiving payment from developers whose agents require access to this high-quality, non-public data. This initiative establishes a free-market economy where information is traded based on its utility and specificity for AI applications, fostering innovation and better-informed AI agents.

Exa demonstrates its capability to combine information from the public web with non-public data sources, enabling more comprehensive and nuanced query results. This feature allows AI agents to access a richer, more diverse set of information than is typically available through traditional search methods.

An example showcased the integration of public web data with proprietary information from SimilarWeb, which is not publicly accessible. This ability to bridge disparate data sets highlights Exa's versatility and its potential to deliver highly specialized information synthesis for complex queries.

Exa's ultimate goal is to achieve what it terms 'perfect information,' which would allow complex queries to be answered with seemingly effortless speed and accuracy. Will Bryk likens this capability to completing a year's worth of research in just one second, underscoring the transformative potential for AI applications.

This level of information access is crucial for supporting the exponential growth of AI agent activity in the future. By providing near-instantaneous and highly precise data, Exa aims to enable AI systems to perform tasks that would otherwise require extensive human effort and time.

Exa aims to provide global access to near-perfect information, particularly in anticipation of high-stakes events like the 2028 election. The company believes that well-informed citizens are vital for sound democratic processes.

This objective is framed as building critical world infrastructure necessary for a society to make informed decisions. By ensuring widespread access to accurate and comprehensive information, Exa hopes to contribute to a more knowledgeable populace during significant political moments.

AI systems are projected to generate thousands of times more search volume than humans, a scale so immense that it cannot be adequately represented on current search graphs. This unprecedented growth indicates that human-centric search models are simply incapable of keeping pace with future demands.

Will Bryk posits that virtually every software interaction will eventually be grounded in some form of search. This exponential increase necessitates a fundamental rethinking of search infrastructure to support the vast and complex information needs of the agentic web.

Will Bryk believes that future societal limitations will stem more from challenges in human coordination than from technological shortcomings. He argues that with increasingly advanced technology, the ability to make sensible collective decisions hinges on effective coordination.

The quality of information consumed directly impacts the effectiveness of social coordination. Therefore, technology's primary role is to provide a robust foundation of accurate data to enable better decision-making and enhance human cooperation on critical issues.