Computer-Use Models to Drive Web Agentification

A growing consensus suggests that AI agents will soon perform more web actions than humans, from clicking buttons to making purchases.

The prevailing theories often predict that the web will adapt to this shift through 'agentification,' primarily via APIs and evolving protocols.

Dhruv Batra, however, disputes the idea that websites will generally provide APIs to accommodate these agents, especially given the current state of web infrastructure.

He differentiates between the 'head' of the web's distribution, which includes popular websites that might offer APIs, and the 'long tail' of less popular sites that will not.

Batra describes the expectation of universal API availability across the web as 'delusional,' pointing out the vast number of sites that lack such infrastructure.

Batra highlights local restaurant websites as a practical example of why structured endpoints, crucial for API-driven interactions, often do not exist.

He illustrates the difference between 'easy mode' websites with plain text menus and 'hard mode' sites where menus are displayed as pixelated JPEG images, making text extraction difficult.

Many small business websites, which comprise a significant portion of the web's long tail, lack accessible text, forcing agents to rely on optical character recognition (OCR) rather than direct API calls for data extraction.

The United States has an estimated 15,000 to 20,000 school districts, and a business seeking to track laptop procurements would ideally want a clean API endpoint.

However, legacy systems in these institutions frequently use scanned PDFs for purchasing solicitations instead of structured, machine-readable text.

Accessing even basic documentation sometimes requires formal Freedom of Information Act requests, leading Batra to argue that expecting these institutions to maintain an MCP (machine-consumable protocol) server is unrealistic and 'off the chart' delusional.

The global web encompasses approximately 200 million active websites and nearly 1 billion total sites, representing a vast and complex ecosystem.

Infrastructure changes within these sectors occur at an extremely slow pace, often taking decades to implement new technologies.

Despite advancements in coding agents, gaining unfettered access to these slow-moving institutions presents a significant challenge, with some professional environments still relying on technologies like fax machines.

Modern websites, such as nba.com, often load content asynchronously after the initial page load, rendering direct HTML parsing ineffective.

The initial HTML frequently contains empty placeholders, meaning a raw scrape will not find the desired data.

Much of the crucial information is embedded within JSON responses from background API calls, while visual cues like 'grayed out' menu items convey status without explicit text labels, complicating logic parsing for agents.

Dhruv Batra emphasizes that on-screen information, such as a 'sold out' status, is typically calculated by rendering scripts rather than existing as pure text.

The browser functions as a rendering engine, akin to a game engine, where the final output is composed of pixels.

Attempting to read source code to predict screen pixels represents a fundamental inversion of the browser's intended process, which prioritizes visual presentation.

Given that the web was designed for human consumption, visual data—pixels—emerges as the ultimate source of truth for understanding web content and state.

While visual interpretation is essential for agents to operate on the web's long tail, models can also employ advanced capabilities like writing JavaScript on demand.

By treating pixels as the authoritative source, an agent can verify the success of an action after injecting custom code, ensuring accuracy.

This hybrid approach facilitates a flexible workflow, enabling agents to click buttons when direct interaction is sufficient and execute code for greater efficiency or complex tasks.

Orchestrator systems can launch multiple 'Navigator' instances in parallel within cloud sandbox environments, scaling agent operations effectively.

Dhruv Batra challenges the notion that progress in computer-use models is slow, presenting evidence of rapid advancement.

He notes that the 'Mind2Web' benchmark is effectively saturated, with the latest model, Navigator N 1.5, achieving 97% accuracy and only eight out of 300 trajectories being incorrect according to human evaluation.

Batra suggests that existing benchmarks should be retired in favor of more challenging tasks that necessitate longer chains of interaction to push the boundaries of agent capabilities.

Critics often raise concerns that browser-based agents are too slow and expensive for high-volume operations.

However, Batra counters this by stating that smaller models can dramatically reduce both latency and operational costs compared to larger, trillion-parameter models.

Data indicates significant cost reductions, with tasks decreasing from approximately $2.30 to 80 cents, which facilitates the scalable deployment of these agents.

Batra dismisses the hypothesis that the web's decades-old infrastructure, built layer upon layer for human consumption, will be replaced by APIs in the short term, calling it a 'fantasy.'

He envisions the future of 'agentifying the web' through agents performing tasks via browser automation, rather than a fundamental re-architecture of websites.

This approach will abstract away the complexities of current web interactions, ultimately creating a functional, 'API-like' experience for agents at a fraction of the cost of a full infrastructural overhaul.