Enterprises Need Agentic Software Factories

The "dark factory" concept envisions a software production system where requirements are the sole input, and fully functional software emerges as the output without human intervention. This idealized state, where automation achieves near-perfect determinism, moves beyond just writing code to encompass systemic infrastructure. Current AI agents are still far from realizing this high level of autonomous, deterministic production, operating primarily on isolated tasks rather than end-to-end software delivery.

Current AI agents typically process code as unstructured characters and letters, limiting their ability to understand its deeper implications. A more advanced approach treats code as structured data, akin to a genome, allowing agents to sequence, curate, and deeply comprehend its intricacies. This shift aims to provide agents with the same level of fidelity and understanding that a compiler possesses, enhancing their capability to reason about and manipulate software effectively.

Providing this rich context enables agents to understand complex problem spaces and make informed decisions, moving beyond generic code generation to context-aware problem-solving.

Currently, AI usage often exists in isolated environments, such as OpenAI or Anthropic platforms, making it difficult to measure its true impact on the Software Development Life Cycle (SDLC). Integrating AI session data into the traditional SDLC is essential to start modeling its effects, asking "what if" questions, and calculating a clear return on investment. This integration enables the collection of rich metadata from agents, providing the foundation for better performance tracking and strategic alignment.

While coding agents excel at reasoning and planning within a single repository, their inherent non-determinism poses significant challenges when scaled across thousands of services or multiple repositories. This non-deterministic behavior creates compounding failure risks, undermining the reliability enterprises demand. To overcome this, organizations require deterministic tooling, exemplified by Moderne's use of compiler-based recipes, to ensure trustworthy execution and predictable outcomes at scale.

Effective governance of AI agents in the SDLC must begin with robust observability and measurement before any optimization can occur. Many organizations currently lack sufficient visibility into the outcomes and impacts of their AI agents, making it difficult to understand their contributions. Establishing comprehensive measurement capabilities is the critical first step to ensuring that AI agent outputs align with broader company goals and to lay the groundwork for informed governance strategies.

To manage the probabilistic nature of AI and move towards semi-deterministic outcomes, complex tasks within the SDLC must be broken down into much smaller, manageable components. For example, a code review, seemingly a single task, actually comprises hundreds of specific subtasks. By applying processes, guardrails, and precise metrics to these smaller components, the stochastic outputs of AI can be refined. This approach suggests a potential to entirely redefine the SDLC, rather than merely automating existing steps.

To effectively assess quality, these abstract goals must be broken down into specific, quantifiable metrics. AI can play a crucial role by aggregating historical data to score aspects like code drift over time, providing qualitative answers about how code evolves. This allows organizations to move beyond subjective evaluations to objective, data-driven assessments of quality.

Many engineering organizations currently measure AI impact through metrics like license utilization or the percentage of generated code, a practice often dubbed "token maxing." However, high token consumption does not inherently guarantee useful software output. The critical shift is from token maxing to "outcome maxing," where AI efforts are aligned with strategic initiatives tracked in project management tools like Jira or Linear. This ensures that AI contributions are evaluated based on their direct impact on business value rather than mere operational volume.

Measuring AI success should shift from assessing what the AI consumes (like tokens or computational resources) to what it actually delivers and lands in production. AI tokens are analogous to electricity for a factory—essential for operation but not the final product itself. Therefore, success should be quantified by code that successfully merges and remains in production, providing tangible value. Frameworks such as DORA or SPACE can be adapted to evaluate the collective impact of human-AI collaboration on shipped value.

A proposed ROI formula balances productivity gains against technical debt and operational costs, creating a clearer picture of AI's net impact. The numerator includes features and infrastructure that accelerate development, representing positive output. The denominator incorporates negative factors like bugs, incidents, human labor costs, and AI operational costs. The goal is to observe a positive trend in this ratio over time, providing a relatively straightforward method for calculating ROI, drawing inspiration from principles found in 'The Phoenix Project'.

The current vision of fully autonomous AI agents, capable of writing, testing, deploying, and fixing software independently, is far removed from the practical realities of enterprise software development. This model is particularly impractical and potentially hazardous in highly regulated sectors like banking, where human oversight is critical. Software delivery remains a collaborative "team sport," requiring ongoing interaction and cooperation between humans and AI agents, rather than full automation.

The role of AI agents is shifting from being direct executors of every task to acting as orchestrators, planning and triggering various tools and processes. This means agents should focus on higher-level decision-making and coordination rather than performing every manual action. Maintaining human logic in the loop at a strategic level is crucial, as is correctly positioning the agent's function within the workflow. This strategic placement is often more vital than the agent's raw code generation speed.

We are still in the early stages of the AI transition for software development, roughly 4-5 years in. In this evolving landscape, the primary role of engineers is that of "direction setters" and stewards, actively guiding and steering AI tools rather than simply deploying and forgetting them. The current era emphasizes stewardship and continuous oversight, reinforcing the idea that software development remains a collaborative human endeavor, even with advanced AI assistance.

One effective strategy for integrating agentic autonomy involves using a "review panel" of agents, specifically designed for each Pull Request (PR), that has been encoded with the tribal knowledge of senior engineers. This panel assesses PRs and categorizes them by risk level (low, medium, or high), determining whether the review can be fully automated. Organizations must identify their own comfort threshold for autonomy, progressively automating areas where risk is manageable and confidence in AI performance is high.

If human code review processes are deficient, this presents a prime opportunity to introduce AI agents to enhance quality and efficiency. The aim is not for agents to be flawless, but to be "good enough," comparable to human drivers who are effective without being perfect. Focusing on these well-defined and verifiable environments maximizes the chances of successful agent adoption and impact.