Why Browser Agents Fail in Production

Running a browser agent demo is straightforward because it typically involves a single site and a single run under watchful eyes, as Derek Meegan observes.

Production environments, however, require thousands of unattended runs, where external factors such as site changes or model inconsistencies significantly complicate deployment and success.

Models process input tokens representing page state, overarching goals, or historical steps to predict and propose the most probable actions, according to Derek Meegan.

Browser interfaces comprise multiple layers, including HTML, accessibility trees, and browser state storage, with interaction strategies spanning textual representations, computer vision through screenshots, and dynamic code execution using JavaScript or the Chrome DevTools Protocol (CDP).

Browser agent trajectories range from purpose-built agents designed for single, specific tasks to highly autonomous just-in-time automations that handle arbitrary user-provided tasks without prior knowledge of the goal.

Incremental cost and failure risks accrue throughout a browser agent's trajectory, given that value realization occurs only upon the terminal completion of a task.

Derek Meegan explains that unlike other systems, web automations offer no partial credit for partially completed steps, making the entire process a binary outcome.

A browser agent achieving 99% accuracy per step succeeds only 36% of the time over 100 steps, exposing the disconnect between local performance and overall task completion.

Derek Meegan highlights that these cumulative errors lead to increasing costs and risks, ultimately resulting in a low probability of realizing the final value, though he asserts that reliable production outcomes are still attainable.

Derek Meegan argues that the equation Profit = Revenue - Costs is underutilized when assessing whether a browser agent's value exceeds the resources it consumes.

Agents should be viewed as a financial line item that must generate more value than they consume, with value defined as the consistent and repeated completion of customer tasks.

According to Derek Meegan, cost and maintainability are engineering optimization problems, and success should be quantified by concrete artifacts like confirmation emails, receipt IDs, or database records rather than abstract metrics.

Derek Meegan demonstrates that an agent with only a 50% success rate per run can achieve a 94% success rate with just four retries per transaction, underscoring that customer satisfaction hinges on workflow completion, not the number of attempts.

Derek Meegan anticipates that model costs will become a diminishing component of deploying browser agent trajectories over time due to the expanding open-source landscape.

However, infrastructure, compute, and integration tooling will remain critical cost drivers, while maintenance demands significant investment in observability, developer time for configuration, and continuous re-evaluation in response to environmental changes.

Enhanced model capabilities enable a more intuitive understanding of user intent and the ability to course-correct effectively when errors arise, ultimately helping to complete the intended goal.

Despite these advancements, web automation faces the challenge of websites rarely offering APIs, forcing agents to handle web interactions indefinitely, which is complicated by the inherent indeterministic nature of models that may deviate from the critical path between runs.

To automate the retrieval of explanation of benefits documents from a health insurance portal, developers must encapsulate complex operations into deterministic tool calls and verify outcomes against a system of record.

To achieve this, complex operations such as downloading and file retrieval are encapsulated into single, deterministic tool calls, while OCR tools verify the agent's work against a system of record.

Authentication logic is extracted from the agent's responsibilities for reuse across different business workflows, and 'skills' are developed as standard operating procedures to guide the agent along the critical path, minimizing ambiguity and improving reliability.

The finalized system architecture employs a serverless authentication function to prepare the browser environment, allowing the agent to follow a pre-defined skill path, use deterministic download functions, and validate results with an OCR tool.

By removing steps that are not essential to the model's core responsibility, the overall system becomes more maintainable and performant, with a primary focus on reducing step counts to overcome inherent reliability limitations of browser agents.