Warp CEO Zach Lloyd on Building a Software Factory: From Slack Request to Pull Request and Measured Agent Quality

The host opens by introducing Warp's CEO Zach Lloyd, saying Warp has pivoted into what the host calls the definitive software factory, and that the platform is open source. The host notes that Lloyd's talk at AIE, which defined the concept, has been going viral and is among the top talks of all time.

The host argues that a software factory is becoming a table-stakes capability, and that buyers will expect features such as cloud hosting, a control plane, basic security, and team features. The episode is released as a preview of AIE Code, the last AIE flagship event of the year, held in San Francisco, where Lloyd will give a keynote on software factories and recap the year.

Asked about the response to his talk, Lloyd says the argument has hit a nerve, because people are very interested in software factories, an area Warp has been focusing on. He says Warp has fairly strong and distinctive takes on what a software factory actually is and what it means for a software developer.

When asked what people think a software factory is, Lloyd says most companies he talks to are trying to build an internal software factory or a set of them. He offers his own definition: an agentic system in the cloud in which you define the roles of various agents across building software for your team or product, with a certain amount of autonomy so that agents can act on their own within the system.

Lloyd says such a system should be highly measurable and closed-loop, so that a team can tune, measure and benchmark it over time and gain confidence that its move to AI-driven development is happening in the most scalable and efficient way.

On common misconceptions, Lloyd says people often think that buying a tool will produce a money machine. He says many people treat a software factory as a loose label for heavy agent use, stringing together automations. He argues it should instead be treated as a specific thing defined in code, with workflows and measurement built in.

Lloyd says Warp's factory platform is built on the idea that a software factory is a real, orchestrable and measurable thing, with one factory per product surface. A product surface could be a mobile app or a server app. The factory operates on a set of repositories and needs a set of skills and MCPs, along with an associated workflow.

That workflow roughly matches what a team is probably already doing to change its codebase, such as triage or specification and implementation. The workflow is defined in code; in Warp's case, a YAML file lists the repositories, MCPs and integrations that can be used. Lloyd describes two personas: product engineers who ship features through a factory, and those who build and maintain the factory itself.

Product engineers use the factory through their existing tools. They can message the factory in Slack, use a task tracker such as Linear, interact through GitHub, or call the factory through an API or MCP. Lloyd's metaphor is that a work item goes into the factory, and when it does, a predefined workflow starts, defined in code.

Lloyd says the key point is that everything goes through one system. Once it is in place, a task started in Slack, Linear, or by an automation such as a server error flows through the same process, and the team gets a real-time view of how its own factories are being used across Warp's internal team. He notes that this view shows the full trace of what everyone is doing with coding agents across any product that has a factory.

Lloyd says that when he talks with companies, most do not have this kind of centralization, because most engineers individually use tools such as Claude Code in a terminal, the Codex app, or similar tools in their own local setups. Companies want to reach a world with centralization, and when they have it, they can analyze how agentic coding works across the company.

He describes several dimensions of such data. One is team output, including traditional DORA metrics such as how long it takes to ship. Another is automation, measured by how many times a human had to act on each merged PR. A third is throughput. Cost is also central: Lloyd says companies need to tune factories with the right models and skills to reduce cost per PR over time, and to slice cost by person, by automation, and by stage of the software development lifecycle.

Lloyd describes scorers as a form of online evaluation. They range from general questions, such as whether agents are too verbose, whether they are efficient, and whether code quality is good, to narrower checks. He says Warp provides a set of out-of-the-box scorers that it considers generically useful. As it works with companies, those companies may add scorers for failure modes they care about, such as whether an agent is making a change that breaks the build. All scorers are defined in code and can be customized.

Lloyd explains that setting up a scorer retroactively grades historical agent conversations. He gives the example of grading for redundant tests, a common failure mode in which agents write tests the codebase does not need while working through a problem and leave them in place, bloating the codebase and wasting tokens. A scorer of this kind uses an LLM as a judge, with a chosen judge model, a prompt for judging, and a sampling rate, because running agents has a cost.

Lloyd describes the result of such scoring as a set of smart agents that look across the whole team's coding sessions and report on how they are going. For example, they may find that some agents are writing too many tests. The output is a record and an understanding of where the team should look to improve the way it uses coding agents.

Lloyd says the tool set in this view is typically used by the CTO or engineering manager, and more commonly by a platform team or a developer-experience team trying to get the most out of tools such as Claude Code or Codex. He adds that the general principle applies to any internal factory: store everything, make everything observable, and measure everything, so that engineering rigor can be applied to how coding agents are used.

Beyond Slack, Lloyd says people set up trigger-based ways of putting work into the factory. Once the scoring view is in place, another loop of agents can look for failures and for scores that are not going well, and then suggest changes to the system to make it perform better. Lloyd calls this self-improvement.

He gives an example from Warp's triage step. Its self-improvement agents found 54 failure cases of one form in which the triage step would have done better had it been able to spawn a sub-agent for its investigation. Because the whole factory is defined as code, the coding agents can make a pull request against the factory definition to try to improve it. Lloyd says the next step, which Warp does not yet have, is to replay the runs and verify that a suggested change would actually have improved them.

Lloyd describes benchmarking within the factory context. A team can add any coding agent run from its system to a benchmark suite, for instance a curated set of runs related to front-end coding. The same tasks can then be replayed against a matrix of configurations. Lloyd says he believes in varying not only the model but also the harness: Warp has its own harness, but also lets users benchmark against Claude Code, Codex, or Pi, with a chosen model.

The output looks similar to published results for SWE-bench or Terminal Bench, but is based on the company's own real runs. Lloyd shows a plot of cost against correctness, where the best position is the upper-left corner, with the lowest cost and highest correctness.

Lloyd uses the benchmark plot to compare models. He says one model, which he calls GPT-5.6, was about 95% correct on the task set but cost about six dollars on average, while GLM 5.3 Flash was 93% correct and cost about two dollars. He notes these models are slightly older because the field is changing fast.

He says the data can feed a model routing strategy, or a harness routing strategy, defined specifically for a company's workflows and codebase. Lloyd describes this as moving from guessing to what he calls factory engineering, in which a team asks how it knows that coding agents are being deployed in the right way and builds infrastructure and tools to answer that question.

Asked what someone should think about when building in this space, Lloyd describes a stack. At the bottom is infrastructure. He argues it is foundational to define the factory declaratively as code, with Terraform as the analogy. The benefits include a versioned view of the process, whether called SDLC, AI DLC or PDLC, the ability to create and test branches, the ability to revert changes, and the added benefit that coding agents can suggest improvements because the definition is in code.

Lloyd says the next layer up is a place to gather all the information that agents produce, so that teams can examine it, look for failure modes, or replay past tasks with different variables to measure and optimize the system. He gives an example of a factory conversation that starts in Slack, continues in Linear and GitHub, includes a code review comment, and results in merged code, which might then cause a server crash that needs diagnosing.

Lloyd says a team needs a way to plug in MCPs to get data into the system. As companies get bigger, they want different access for different parts of the organization, such as which MCPs marketing, sales and engineering can use, which adds complexity. He says a compute layer should be treated as a separate component.

Many agent sandbox providers now exist, so Lloyd recommends making a third-party choice pluggable. Warp uses Namespace as its primary provider and says it should still be pluggable, naming Daytona, Modal and E2B as alternatives. He explains that by default a Linux-type sandbox is likely sufficient, but computer use requires more: Warp tests iOS apps by running a Mac VM that runs the simulator and clicks around.

Lloyd says Warp has a very open approach to inference, consistent with its ethos of being flexible. Many companies already have commitments with providers such as Anthropic, OpenAI or Google that they want to use up. Warp lets customers plug in their own inference provider or buy inference through Warp, and lets them plug in their own harness, orchestrating tools such as Pi or any ACP-compatible harness.

Lloyd then gives Warp's figures. When Warp first started using the factory approach, it cost about $80 per PR, because it was using Opus where it should not have, its triage agent used a very high-powered model, it did redundant work, and it had too many sub-agents relative to fewer agents. After internal tuning and testing with built-in features, the cost per PR fell to about $15–20. Over the last month, PR throughput rose 40%, and the revert rate has not increased. Lloyd says this gives him confidence in an approach of observing, measuring and tuning.

Lloyd says the orchestrator is a piece of the platform, not the whole. His philosophy is to bring the factory into a company's existing knowledge-work tools and make the whole system API-driven and MCP-driven, so that the factory works as headless infrastructure with many entry points. He says his stack-based view resonates with companies as a new layer in the developer infrastructure stack, sitting one level back from CI/CD.

Asked who the stack is for, Lloyd says it resonates most with companies at some scale. He says it is overkill for a few people, who should just use a subscription such as Claude Code or Codex and go faster. He says the threshold where it begins to make sense is around 5 to 10 engineers, and the larger the company, the more implementation work it requires.

Lloyd says that because the factory works inside multiplayer environments like Slack and Linear, teams gain benefits from a central system of record. Through the Slack integration, for example, information from a server-crash channel or a user-feedback channel can feed directly into the factory to do the work. The factory can loop in several people depending on who knows the relevant part of the codebase.

Lloyd says the approach does not require a very large company, but it does not make sense for a small team. He says it makes most sense for a company that wants to put in place infrastructure that can keep being built on as models and agents change. He names a poor fit: companies with very bespoke infrastructure and large internal developer teams, such as Uber or Ramp, which he says are building their own systems.

Lloyd says that, in his view, a company should not be in the pure coding agent business, because one cannot build a competitive coding agent on top of a product that costs so much more. The host notes that a $200 plan provides about $8,000 of API usage, and Lloyd agrees it is not money to sneeze at. He says Cursor did this well and then had what he calls a perfect exit by becoming a model lab, once it fully understood the economics.

Lloyd says Warp's factory product is not pitched as a token-reselling product, but as a coding agent optimization platform and a coding agent productivity platform. He lists what he considers the remaining independent coding agents: Cognition, OpenCode and Factory.

Lloyd says his next talk will focus on what it means to be an engineer as factory engineering grows. It will ask what roles and jobs look like, contrast product engineering with factory engineering, and examine what this emerging discipline looks like in practice, including the tools and skills engineers need to build the system that builds software rather than building software directly.

Closing the interview, Lloyd points listeners to warp.dev, where most of Warp's material is available, and says Warp is hiring engineers, go-to-market staff, data team members and a designer. He says he posts content about factories on his Twitter account, @zachlloyd.