GEICO Accelerates Legacy Code Modernization with AI-Native War Rooms

GEICO has observed a significant acceleration in the 'inner loop' of its Software Development Life Cycle (SDLC) due to advancements in AI, particularly in the coding and intellectual work phases. However, according to Ford Prior, the company has found itself constrained by the 'outer loop,' which encompasses all non-coding activities such as decision-making, approvals, and context gathering.

As a risk management company rather than a tech company, GEICO's processes inherently integrate risk aversion, impacting the speed of software development and delivery. The strategic objective is to align the outer loop's pace with the accelerated inner loop, aiming to liberate development from traditional constraints and empower teams to achieve similar efficiencies.

Approximately six months prior to the presentation, an analysis of GEICO's backlogs revealed that engineers were spending a substantial portion of their time on minor updates. After completing patches and minor upgrades, engineers had only 25% or less of their bandwidth available for major changes (V1 to V3 level) that are crucial for maintaining the company's competitive edge.

To address this imbalance, GEICO initiated strategic investments in managed AI agents. These agents are designed to handle the entire software lifecycle for minor work, from specification to deployment, using various harnesses and models. The goal is to free up GEICO's engineers, allowing them to concentrate on significant, impactful projects that they find more engaging and vital for competitive advantage.

GEICO identified that major development work was being bottlenecked by an inability to effectively collaborate and bring together the right decision-makers at opportune moments. This challenge was compounded by a lack of integrated intelligence needed to weave together relevant variables, perform fact-checking, and crunch code to rapidly produce insights for decision-making.

To overcome these obstacles, GEICO established a multiplayer AI-native war room. This war room is composed of key decision-makers, subject matter experts pertinent to the topic, and a dedicated 'driver' responsible for operating advanced AI tools.

The war room features an AI-native productivity suite that integrates communication channels for discussion, interactive design studios, and code repositories. These components are all interconnected with workflows, agent skills, and enterprise Model Context Protocols (MCPs) to enable fleets of agents to perform heavy lifting, including fact-checking and synthesis.

The core objective of GEICO's AI-native war room experiment was to accelerate the delivery of agent-callable APIs for legacy core capabilities. Ford Prior placed a significant bet: that with a budget of $1,000 in AI tokens and one day, the team could map one domain end-to-end, including graph representations of business topology and complex legacy system code.

The project's scope involved converting millions of lines of legacy code from GEICO's claims systems into agent-callable APIs, with the ultimate deliverable being a working demo for leadership by 4 p.m. This experiment was conducted three times across different code modernization missions to validate its effectiveness.

GEICO conducted three distinct code modernization experiments leveraging the AI-native war room, categorized as uplift, transform, and reimagine, based on Anthropic's code modernization plugin taxonomy.

The first experiment, 'uplift,' focused on the rentals domain, which handles rental car logistics after accidents. This involved implementing a newer version of existing technology and extending an existing API. The agent, through an 'interesting process' of data distillation, identified a way to save 8,500 calls per month by providing customers with up-to-date rental information.

The second experiment, 'transform,' addressed the damages domain, which involves assigning dollar values to reported losses. For this, GEICO built a set of new agent-callable APIs to enable a larger workflow that required damages data. The third and arguably most complex experiment, 'reimagine,' tackled the salvage domain, which deals with vehicles declared total losses. This effort resulted in a fully vetted migration plan with API specifications and design mockups, which was ratified and green-lit by product and tech leadership at all levels. Notably, the salvage work also uncovered a production bug, which the speaker found 'shocking.'

The architecture of the AI-native war room combined external vendor tools with GEICO's internal context and methods. On the vendor side, components included a model, a harness, FleetKit, and a code modernization plugin, which are replaceable.

Conversely, the irreplaceable GEICO-specific elements included a vast knowledge base with software standards and historical decision archaeology, a list of tools accessible by the agents on the driver's machine with instructions for their use, and proprietary methods—highly optimized, domain-specific ways GEICO develops software. Critical to the system were judgment protocols, which defined who could make what decisions at specific gates, involving both subject matter experts and leadership. Due to the high operational costs of AI at scale, a 'PhD level' fleet manager was employed to manage the system.

FleetKit is a tightly structured collection of rules, protocols, and instructions, all written in Markdown. It is specifically optimized for the Claude harness, particularly Fable, and is engineered for the core task of architecture transformation. This advanced steering mechanism operates one level above standard system prompting.

The speaker noted that such a system requires advanced models, suggesting Opus 48 and beyond, to effectively manage complex steering. FleetKit functions by taking a high-level goal, breaking it down into smaller tasks, and dispatching agents to conduct research and propose approaches.

The system incorporates a 'human-in-the-loop' queue, which can use tools like GitHub issues, where humans can comment on and approve proposed actions, or other agents can add more context. Once approvals are secured, builder agents are dispatched to implement the changes, often utilizing 'beads' to help agents track everything within their workspace. The ultimate aim is to produce a 'material change,' such as a change request or a code change in repositories.

GEICO's ultimate vision for this AI-native approach is to redefine software development. The goal is for software engineering to resemble work in a factory more than creative play on a canvas. This factory-like process culminates in a 'material change,' which could be a change request into systems or a direct code change in repositories.

To facilitate this, tools like 'beads' are employed to help agents maintain state and keep track of all tasks within their own agent workspace, ensuring a structured and efficient progression through the development lifecycle.

The operational process for the AI war room begins with agents booting up in a cloud code session via a plugin and being assigned a mission. The agent then creates or adopts a Slack channel (or similar communication platform), which serves as the central hub for all war room members and relevant stakeholders to communicate.

As the agent progresses through its nine-step process, it creates two primary types of artifacts at each milestone. It generates well-structured HTML worksheets using cloud artifacts, which can be filled out by humans and are automatically saved to bridge information gaps. Additionally, it produces design documents, including complex architectural visualizations that did not previously exist, which teams were delighted to retain and could be edited by humans and re-read by the agent.

A key feature is the agent's ability to proactively ping relevant human owners from the roster if specific product or business details are missing. Ford Prior noted that the generated architecture diagrams were particularly valuable, often mapping previously undocumented edge cases or systems. The experience in a physical war room in Richmond, Virginia, would typically involve people huddled around a large screen displaying either a Claude code session or these product worksheets and design artifacts.

The execution environment of the AI-native war room leveraged a combination of cloud resources for intensive data analysis and local agent operations. Cloud-based code crunching and graphing of legacy domain-specific languages were performed in the cloud.

Simultaneously, local agents ran on the driver's laptop, gaining access to all Model Context Protocols (MCPs) on that machine. Many of these MCPs were not yet in production but were highly optimized for GEICO's specific problem space, allowing agents to surface valuable data, including chunked versions of codebases. A dedicated bindings file on the laptop described the machine's configuration and how it mapped to the FleetKit framework, enabling the local agents to interact with these tools effectively.

The modernization process adopted a structured nine-step workflow, which was largely forked and adapted from a code modernization plugin. This plugin formed the core of the entire process.

The project visualization prominently featured orange-colored elements, which represented worksheets requiring human approval. Ford Prior noted that human input was significantly higher at the beginning of the modernization journey, gradually decreasing as the process advanced. An interesting detail of the code modernization plugin is its deterministic analysis, verifying that each preceding step successfully created all necessary artifacts, including files.

GEICO gained several operational lessons from its AI-native war room experiments. Ford Prior cautioned against configuring agents to set up recursive cron jobs, as this can lead to repeated session launches and rapid depletion of budgets. The team also observed that prompt caching delivered higher utility than initially anticipated.

Benchmarking tests, using recently written existing code as 'golden data,' indicated that Claude Sonnet 3.5 performed just as well as, if not better than, Fable in writing complex legacy code after context building and agent dispatch. This suggests the importance of model selection for achieving optimal code quality. Future enhancements are planned, including improved gate enforcement and updates to decision models within the harness and model selection framework.

Looking ahead, GEICO aims to improve its context management, as the current system relies on simple markdown files without advanced graphing or search capabilities. Ford Prior expressed a need for more optimized ways to search and review the extensive context used by the war room.

Another critical improvement involves treating audio as a data source. The goal is to automatically transcribe conversations from meetings to avoid manual note-taking and minimize information loss, which can be distracting and incomplete. Ultimately, GEICO is working to strategically deploy this AI war room across the organization, focusing on how to integrate human engineers most effectively. The objective is to position the brightest engineers to 'pick the next hill' for optimization, guiding agents as they climb the initial ones, thereby ensuring human expertise directs the continuous improvement efforts of the AI agents.