Warp's AI Factory Self-Improves with Skills, Memory, Routing

Suraj Gupta leads harness development at Warp, a company that began as an agentic development environment centered around a modern terminal.

Warp is now expanding its focus to build a comprehensive cloud agent platform, aiming to facilitate the creation and operation of advanced software factories.

Engineering teams are transitioning from building products to building and maintaining end-to-end factories that manage processes from issue triage to production deployment.

The core challenge lies in enabling these factories to self-improve over time, becoming more efficient and faster, a goal that redefines the role of engineers from product builders to factory architects and maintainers.

Skills function as procedural memory for agents, which can deteriorate as operational environments change, necessitating continuous refinement.

An 'inner loop agent' is primarily responsible for executing specific tasks, such as triaging issues.

An 'outer loop agent' observes the performance of the inner loop agent, analyzes its mistakes, and incorporates human feedback to continually improve its skills, ensuring they remain relevant and current.

Warp utilizes an internal factory to manage its own client repository, which has been open-sourced.

This factory includes a triage agent that automatically processes new GitHub issues, identifying missing information or potential duplicates and creating a repeatable workflow triggered by each new submission.

The outer loop agent actively monitors the decisions made by the inner loop triage agent, gathering feedback signals such as user thumbs up/down reactions and comments from users or Warp employees.

This feedback is synthesized to periodically update the inner loop skill, enhancing its effectiveness.

All updates are meticulously tracked via Git pull requests to ensure full observability and accountability, with human review being a mandatory step to prevent any regression in agent performance.

Agents frequently expend unnecessary tokens by re-gathering context for issues they have previously addressed.

To combat this, persistent memory functions as a fact store, specifically scoped to an individual agent, retaining crucial information.

Similar to skill improvement, an outer loop agent extracts learnings and outcomes from the inner loop agent's operations, populating this memory store, which humans can version, edit, or delete for precise control, and agents can trace specific memories back to their original runs for verification.

The Sentry agent effectively uses its repository of stored memories when triaging new issues.

It explicitly reports which specific memories were accessed and utilized during its decision-making process, ensuring transparency and preventing the agent from becoming stuck in local optima during its analysis.

Warp Zone and proprietary systems benefit from persistent memory that automatically generates knowledge while allowing users to maintain full control to review, edit, version, or delete stored data for traceability.

Memories are automatically generated for users, who retain full control to review, edit, version, or delete these stored memories, ensuring complete traceability and oversight.

High costs arise when using complex models like Opus for simple triage or CI tasks, which is why Warp provides 'auto models' and custom configuration files to optimize routing for Pareto efficiency.

Warp addresses this by offering 'auto models' that evaluate Pareto efficiency, guiding users to select the most optimal models for specific tasks, and enables users to define custom routing rules through configuration files, assigning particular tasks to appropriate models.

For example, database migrations can be routed to GLM, while documentation tasks are directed to Quen, ensuring cost-effective and efficient resource allocation.

Currently, rule-based routing is more of an art than a science, but Warp is developing customer-facing evals to allow users to define specific criteria and configurations.

This will enable automated model routing based on which models perform best for particular, user-defined workflows, with insights integrated into the product after internal 'best-at-k' testing identifies efficient task-model pairings.

Warp is actively developing customer-facing evaluation frameworks to move beyond generic online benchmarks, allowing users to define metrics precisely relevant to their unique software issues.

This system facilitates direct comparisons between different AI models, such as Haiku, GLM, and Opus, analyzing their performance based on custom configuration parameters and specific workflows.

Warp utilizes an 'eval sidecar' to determine the optimal model for specific tasks, employing a 'best-at-k' approach by simultaneously running multiple agents across various models.

Internal testing has revealed that GLM is highly effective for UI-related tasks, demonstrating greater efficiency compared to using Opus, and the team plans to integrate this sophisticated routing capability directly into the Warp product for broader customer utilization.