Why AI Makes Individuals Faster but Not Companies: Opinionated, Simple, Reliable Agents

Troyanovsky introduced himself as one of the two co-founders of Basis, based in New York City. He said the company's central question is how to get agents to do production-scale work reliably enough that accounting firms across the country trust them and use them end to end.

He said much of the company's research is organized around how to get agents to do real work in the real economy.

Troyanovsky began with a question for the audience. Anyone watching the talk has probably increased their own productivity a lot, using tools such as Co-work, Codex, or Claude Code. He said that his own productivity may have gone up about 100x.

Yet, he said, if one looks at the average company, the average enterprise, or the average firm, productivity has not gone up 10x or 100x. If he were working inside a company, he said, the gain would not be anywhere near that order of magnitude. He asked why that is.

Troyanovsky said that at Basis one of the company's premises is that humans are already used to working with non-deterministic systems; the difference is that those systems are normally their co-workers, not their computers. To answer the productivity question, he suggested looking at how existing intelligent systems, meaning companies, are organized.

He described the problem as one of trust. Organizations need to trust that people at every level are doing their work in a way that will produce good outputs consistently at scale. He used an accounting firm as an illustration, though he said it could be any traditional company. The head of the firm, he said, has no idea of the exact day-to-day of the people at the bottom, yet trusts that the work will be done.

According to him, that trust does not come from deterministic systems, because people are not deterministic; they are intelligent entities. Instead, organizations put processes in place. If the people who designed those processes believe they will generalize to good outcomes, they can be comfortable that, even though the people executing them are not deterministic, following the process will generally produce good outcomes.

He said this holds even for the smallest steps. In the monthly accounting work for a client, reconciling cash requires bank data, and even that has a process. The process could be very basic, such as using approved software to extract the data, but a process is still put in place. He noted that a firm would not expect each employee to extract PDFs however they liked, because then there would be no process.

Troyanovsky said that everyone already has tools such as Co-work, ChatGPT, Codex, and Claude Code, so the question is where the productivity is. He said Basis's view is that there are two main problems with what he called coding-style agent products, in which the user works with an agent and simply tells it to do something.

The first is a competency gap. He argued that agents, as they are generally thought of today, outsource complexity to the user. The user has to manage skills and memories and decide when to use the agent, which requires a kind of theory of mind about the system. He said this is hard, and while the audience may find it easy, for the wide majority of knowledge workers he expects they have no idea how to do any of it.

The second is a process gap. If the user is told, in effect, here is your agent, do what you want, then the way the work is done depends on whoever the user is and how they customize the agent. He said this goes against the way organizations scale. The process should come from senior leadership down, not from an individual employee deciding how to do their job. Returning to the PDF example, he said that if every employee made up their own way of extracting data, that would not be a good outcome even back in 2018.

Troyanovsky said that Basis strongly believes the key to building production agents that do work end to end is solving these two problems. The consistency gap is solved by making the agent simple. A user should not even have to know it is an agent, or should not really feel it. It should be as simple as working with a co-worker: you tell it what you want, and everything you expect to work should work.

The process gap is solved by reliability. He said the goal is not having a stroke of genius one day and forgetting something the next. He stressed that the point is not perfection, since humans are not perfect and nobody is looking for it. What is wanted is reliability, consistency, and trustworthiness.

He explained that being simple and reliable means the user does not have to think about what to do, and the agent can do it without the user knowing its internals. Doing things the same way consistently requires being opinionated. He added that getting the outcome right is not the main task, because the point is to handle work in a similar way. It is acceptable to be unable to do something, but one should consistently be unable to do it and then improve, as a person does. An expert who suddenly misses something basic is not reliable.

Troyanovsky gave an example of an opinion. Suppose an agent needs to read a PDF. There are many ways to do it: send it to a VLM, write Python code and use an open-source OCR library, use one of several OCR-specific model providers, run OCR and have a VLM check the work, have five VLMs check it, or run 100 VLMs and take the best result. He described the possibilities as literally infinite.

If the agent is simply told to extract a PDF, he said, it might produce good outcomes, but they will not be reliable. He said this is not because the process it followed was bad, but because the process it followed was not opinionated.

He said opinions come from initial intuition and from expertise. For tax work, he said, he knows that if an agent reads the primary source before returning an answer to the user, it is more likely to be accurate, and he does not need a set of evals to tell him that. Similarly, if an agent creates a PowerPoint and takes a picture of it before returning it, it is less likely to have a formatting problem, which he called obvious.

He emphasized that opinions are not only about accuracy. Taking a picture of a PowerPoint adds latency, which may not be wanted. If the agent makes formatting mistakes only one time in a hundred and that is acceptable, then the picture step is unnecessary. He said the point is not exactly what the opinion is; the talk would focus on why opinions matter and how to use them at scale in production, not on how to develop them.

Troyanovsky asked how one knows that outcome-based evals generalize to production. He defined outcome-based evals as what people traditionally call evals: a case has a right answer, and there is a rubric for judging what the agent did. A set might contain a thousand or a hundred cases, but in production, doing real work, there may be hundreds of thousands of situations.

Troyanovsky argued that this number will grow as trajectories get much longer, because the number of decisions made within a trajectory grows exponentially as agents' long-horizon abilities increase. He added that one cannot prove every trajectory in the distribution is correct, because there is no ground truth for each trajectory.

He suggested, however, that one can look at the trajectories to see whether the behavior one has an opinion about is actually happening. If it is happening, one can infer that it is more likely to generalize to production. He acknowledged that this is not perfect and that one does not know for sure, but at least one is having an opinion.

Troyanovsky gave an accounting example. If there is a mistake in a bank reconciliation, he said, he does not want the agent to post an entry to make it tie. He wants it to explain the difference instead of putting a plug. He admitted this is subjective and that someone might want the other approach.

What he does not want, he said, is for a customer to get one result one time and a different result another time.

Asked what he would do, Troyanovsky said the first step is to understand what governs the behavior. He said behavior is not produced simply by writing two sentences into the system prompt and having it work. That might work for very simple, narrow agents. In large agent systems, behavior is a combination of the system prompt, progressive disclosures, possibly tool descriptions, and the training of the model.

He offered a tip for building production agents: think of the context the agent will see while working as runtime training data. He said the word prompt trivializes this a bit, since the context is effectively the data the agent is trained on at runtime, not literally but in effect. One should therefore care about every single token and understand how that data causes the agent to behave in certain ways.

He said that when putting behaviors into production it may be very difficult to put one behavior into exactly one paragraph, because the behavior comes from all of the data at once.

He said one of the things Basis developed is the concept of a behavior spec. The idea is to state, before updating or writing the context, what is actually wanted and what the intent is. He described it as a product spec for the agent era.

He said one should not write down every detail, but should take the most important opinions and record them, because writing everything duplicates state. The spec can then be used to align the team, and also as a rubric. For a given trajectory without a ground truth, one can ask whether the agent got into a situation where the behavior was relevant, and if so, whether it actually exhibited the behavior.

He argued this lets one scale the signal from trajectories far more than increasing outcome-based evals, ground truths, and rubrics. Those are important, he said, but they cannot be scaled nearly as fast as the ability to supervise the agent's behaviors.

Troyanovsky said he would give a quick example and that this is the second concept, called forking, which can be combined with behavior specs. If one has used Claude Code or the Codex web apps, there is a fork icon under each message that creates a copy of the agent to continue from that point. This works well on a local computer. He said building a system that does this in the cloud perfectly is hard, because the database environment and the prompts all have to be recreated.

He described a flow. A trajectory, which could be in an eval set, in testing, or in production, fails the behavior eval. One wants to know why it did not exhibit the behavior. One goes into the trajectory and goes to just before the point where the agent made the decision that contradicted the behavior, assuming the behavior is good and actually producing signal. Then one forks it.

After forking, Troyanovsky said, one can ask the agent right after it made the decision why it did that. Alternatively, one can update the agent's context in the fork and see whether that improves the behavior. With forking, he said, one can start to take the signal and figure out how to improve the harness to leverage it.

By contrast, if one looks only at signal from an outcome-based eval, it is much harder, because the run failed the outcome and one may not know why. He noted that a single agent trajectory run of five hours can involve potentially a thousand inference calls.

Troyanovsky said his favorite tool, which he admitted sounds silly but which he promised is very effective, is simulations. When working with coding agents, one can ask the coding agent to make a sub-agent that pretends to be the production agent. One must tell it not to cheat, provide the system prompt and skills, and render everything accurately. One can then ask what it would do in a given scenario, supplying the situation to see whether the situation seen in an eval or in production can be recreated.

If it can be recreated, one can tweak the context and see whether the sub-agent changes its behavior in response. He said this is far less faithful than forking, but it is much cheaper and faster, which lets an agent builder iterate on ideas without the expensive process of forking an entire environment. He said anyone can do this today without building anything, and that he did it for a year and a half or more before his team built forking into its internal systems.

Troyanovsky said the main idea is understanding whether the agent generalizes to the distribution of situations it faces. One cannot know that purely from the outcome eval. One can try to choose the best outcome evals to represent that distribution, but one does not really know. Supervising the process through methods like these gives additional signal about whether the agent generalizes properly.

He said that as more signal comes in, one can start to close the loop, and that an opinion is what makes this possible. Without opinions about the system, one cannot develop that signal.

Returning to the firm analogy, Troyanovsky said an opinionated agent can be more reliable and can also be simpler. It may start by doing something narrow, but because people work with it, they see it is easy to use and reliable. As it gets better, they trust it more, just as with a co-worker.

He compared this to how people judge a pull request. When one receives a PR, one looks at two axes: how risky the change is, and one's perception of how competent the author is. If the least competent person in the company sends a PR editing a core service, one will look at it closely. If the most competent person sends a PR editing an internal document, one may not look at it as much. He said the same holds for humans working with agents as for humans working with humans.

Troyanovsky anticipated an objection: if agents are allowed to do what they want, won't they find a better way and move beyond the work as designed? He acknowledged they might, but argued that this is not how one scales GDP or output. Nobody wants an agent to move beyond the work and change how it handles their tax return. People want the work done reliably, at scale, in a way they can trust and that their people know how to work with.

He closed by saying that for agents to diffuse, they need to be simple, and for teams to trust them, they need to be reliable. Getting to simple and reliable means developing opinions about how the agent behaves, which in turn allows building the additional signal needed to close the loop and achieve self-improving agents in production.