Kimchi Harness Optimizes AI Costs

Enterprises are encountering rapidly expanding AI expenses, with one Indian company reportedly spending $500 million on Anthropic services in a single month.

Uber's CTO also noted that his team exhausted an entire year's Anthropic budget within just four months, prompting many organizations to impose stringent token limits on developers to control these escalating costs.

Gemini 1.5 Flash is priced at $3.5 per million tokens, yet it incurs a $75 expense for a single task, a discrepancy researchers noted is emblematic of why token-based budgeting fails to capture actual operational costs.

For instance, Gemini 3 Flash is priced at a seemingly low $3.5 per million tokens but incurs a high $75 per task, while Minimax 2.7, at $1.5 per million tokens, costs an even higher $148 per task for the same quality output, showing that raw token price does not equate to operational expense.

Kimchi, an automated harness developed by Cast AI, dynamically selects the most appropriate AI model for specific tasks at the optimal time, based on the task's outcome, leading to improved efficiency.

Over three months, this system achieved a 2.5x reduction in total coding costs, even as total token usage increased by 1.5x, demonstrating its ability to enable 'unlimited' development through background optimization.

The Kimchi harness automatically adjusts its preferred models in response to real-time changes in task performance, demonstrating its dynamic optimization capabilities.

A notable shift occurred on June 12th, where Kimchi 2.6 was the most popular choice on June 3rd, but by June 21st, Minimax 3 had become the top-performing and most selected model, illustrating the system's ability to identify and switch to more efficient models faster than human intervention.

Autonomous coding agents bear the responsibility of optimizing cost management to provide developers with the most cost-effective tokens while maintaining high output quality.

The Kimchi framework, an open-source solution, addresses these needs by ensuring that developers can access efficient resources without being constrained by budget limitations.

Cast AI developed the Kimchi harness approximately six months ago after experiencing escalating cloud bills from their own coding operations.

The open-source platform utilizes the Pono SDK to automate long-running tasks, typically those occurring every two to three hours, by selecting the most efficient model for each process.

Engineers utilize a scoring system within the 'Ferment' process to evaluate quality at every stage, as it breaks tasks into milestones and executes them autonomously for two hours or more.

The platform integrates a scoring system to assess the quality of output at each stage, ensuring that tasks meet predetermined standards, and employs a higher-order model to re-run tasks if initial quality is insufficient.

This iterative optimization ensures that token usage is precisely aligned with the required quality of the final outcome, preventing unnecessary expenditures on over-specified tasks.

Artifacts from the Kimchi process are considered complete only if they achieve at least a 'B' score, with an option to aim for an 'A' grade, indicating a robust quality control mechanism.

Deployment is currently restricted to the staging environment to maintain a 'human-in-the-loop' safeguard, reflecting a design choice to ensure human review before code moves to production, with future plans for automated production deployment based on observability and health checks.

Agentic engineering generates extensive code changes that have become increasingly difficult to review, especially as less technical personnel are integrated into development pipelines.

Code reviews now demand an understanding of the intent and specifications provided to the agent, rather than simply analyzing lines of code, a complexity addressed by Kimchi Teleport, which enables developers to run secure sandboxes in both SaaS and on-premise environments.

Teleport by Kimchi offloads coding tasks to remote environments hosted within hyperscalers such as Google Cloud, synchronizing the entire development environment with a Kubernetes container that operates continuously, even if the local laptop is closed or offline.

This system ensures proper resource sizing and security compliance for coding sessions and only interrupts the developer when human input or clarification of specifications is required, facilitating uninterrupted workflow.

Approximately 62% of engineers at Cast AI now exclusively use Teleport for their coding tasks, highlighting its significant adoption rate.

This high usage is attributed to its ability to allow engineers to maintain coding progress during commutes, holidays, or while offline, enhancing overall productivity and flexibility.

Engineers, product managers, and colleagues review work-in-progress within Kimchi Studio, a collaborative interface that prevents wasted effort by identifying incorrect specifications early in the development lifecycle.

This platform enables engineers, product managers, and other stakeholders to review work-in-progress generated by the Kimchi harness, preventing wasted effort by identifying incorrect requirements or specifications early in the development lifecycle.

It features a Kanban-style interface for managing session status, reviews, and backlogs, providing a centralized, graphical dashboard to visualize and share coding sessions across the team.

The core Kimchi harness is entirely open source, making its model optimization capabilities accessible to the broader developer community.

However, Kimchi Studio and Teleport require a Google account for deploying their cloud-hosted container environments, with a setup process that takes approximately five minutes.