AI Agents Enter an Era of 'Superhuman' Computer Use, Opening New Possibilities

AI agents' computer-use capability has just entered the practical stage and is said to be very close to superhuman level. By operating the screen directly, they can connect to nearly any software without APIs or connectors.

As model speed increases and costs fall, large-scale operating costs are also trending down, which is expected to positively affect the spread of AI agents.

These advances are expected to play an important role in creating new consumer experiences and strengthening enterprise security and auditability.

In enterprise settings, safety and auditability emerge as key factors when using agents. In particular, the ability to control the degree of human involvement and the work records is highly valued.

For legacy software, tools such as MCP often do not exist, so screen-operating agents can become the only automation route.

In the consumer domain, the main use case is delegating tedious everyday tasks such as booking and purchasing to an agent.

Dot is a proactive agent that has its own cloud computer and can use a full web browser to carry out various tasks on the user's behalf, such as purchases and haircut bookings.

Examples of practical work include detecting invitation emails and alerting the user to events missing from the calendar, or periodically reviewing Slack messages to organize and report feedback.

Such agents are said to be especially useful for long-running, proactive tasks that do not always proceed in the same form.

AI agent developers are adopting a strategy of offering both fast, powerful models and inexpensive ones. Over time, choosing the right model for each type of task is expected to become even more pronounced.

Cheaper models are also continuously improving, creating an environment where users can flexibly choose models based on the speed and cost efficiency each task requires.

As AI agents become faster, enterprises may face a stage where they cannot monitor and audit every action. Accordingly, deciding what the AI agent can do, which tasks it can access, and where humans intervene becomes a key issue.

When developing Astra, enterprise control features were introduced so customers could choose the level of human involvement. This is implemented so that human approval is required when the agent downloads or uploads files, or visits a new website.

The position was presented that humans must intervene in sensitive tasks, and that approval should depend on the type and sensitivity of the task.

Computer-use agents are described as general-purpose tools that do on-screen work in place of people. The speaker mentioned the habit of first asking whether the agent can perform any task they themselves can do.

In the past, model capability was lacking, but recently it has advanced to the point where agents can handle most tasks a person can. For ambiguous requests, the agent proceeds by asking for additional information to fill in context.

Computer use is employed like a general-purpose interface that works across all apps. Instead of building separate app-specific connectors like MCP servers, the agent directly operates the same screen a person uses.

In this way, the agent has the advantage of being able to work with nearly any software without a connection tailored to a specific app.

Even if a model is well aligned and resistant to prompt injection, enterprise administrators still want to directly restrict which apps and websites the agent may use. Two layers of control were proposed for this.

The first layer is building a sandbox around the agent so it can only use specific apps or sites. The second layer requires human approval at moments of potential risk, such as file downloads, uploads, and visits to new sites.

The greatest opportunity in adopting AI agents is automating tedious, repetitive workflows. Work in large systems such as ERP (enterprise resource planning) was cited as a representative example.

Such automation is tied to security, auditing, and approval controls, so actual deployment is expected to vary with each company's context.

Sales teams use a variety of software to obtain transcripts and data. An AI agent can perform prospecting using a computer and automatically enter the results into a CRM (customer relationship management) system or other outreach tools.

This is expected to improve efficiency by automating the repetitive, time-consuming work of data collection and entry in sales activities.

Talking to computers by voice is emerging as a natural interface, and a view was presented that software forms may change over time. The perspective is that existing user interfaces (UIs) will not disappear entirely; rather, a new layer where agents operate existing software will emerge alongside them.

One presenter, emphasizing the value of traditional UI design in their own work, pointed out that people still want to understand and manipulate things with their own hands.

In response to concerns that AI agents may communicate among themselves until humans no longer create anything directly, one presenter argued that the value of craft and hands-on creation will remain.

Citing their experience of building board-game strategy coaching software for their spouse, they stressed that a beautiful world in which individuals create software tailored to themselves will continue.

Consumer apps emphasize the enjoyment of use and the value of learning, while in B2B apps, tracking features such as audit logs and human control are core requirements.

In particular, in B2B settings agents must accurately apply rules such as tracking the reason for payments and the 30-day refund policy, and it was suggested that a certain level of control and user interface is still needed for this.

In discussing how AI agents and humans should collaborate, concerns were raised about the appropriate timing and manner for humans to intervene in the work loop.

Features are being considered where the agent notifies the user when something goes wrong or at important completion points, but summarizes the information so the notifications are not disruptive. It is important that users can trust AI as a tool and communicate with it effectively.

AI agent developers are focusing on raising computer-use speed to superhuman levels and strengthening real-time collaboration between humans and AI. The goal is to integrate various UI interfaces to provide a more fluid computer-work environment.

Most current computer-use products are background-centered, but going forward developers plan to build an environment where users and agents interact in real time through a wider range of interfaces.

For consumer products, the goal is to improve agent reliability to make real-world use more convenient and, ultimately, to let all users use computers more efficiently and enjoyably.

Past AI agent products such as 'Alice' were evaluated as having low adoption because users perceived them as ordinary browsers. However, as model capabilities have improved significantly recently, usage of browser-based agents is growing rapidly on a weekly basis.

Developers recognize that users of current computer-use features are only a fraction of their overall base, and they aim to accelerate market growth by securing broader accessibility beyond desktop, including mobile.