The Rediscovery of Harness: A Strategy for Maximizing AI Performance

For a while, harness was considered simple prompt engineering or a wrapper, with low research value. However, this perception is changing as the practical impact of harness improvements on model performance has been proven. A mere transition from harness version 1 to version 2 resulted in an 18% performance improvement, which was a critical factor in the success of Arcade AI.

This suggests that harness is not just a simple tool to assist specific model tasks, but a key research area that maximizes the potential of models and enhances their real-world problem-solving abilities.

Currently, AI model intelligence is primarily measured based on perplexity, but this method has limitations as it does not fully reflect a model's adaptability to real-world domains. Even if a model shows high performance in specific test environments, it struggles to quickly adapt when facing new domains or unpredictable situations.

This indicates that while a model can 'measure' intelligence through extensive testing experience, its ability to 'adapt' to diverse real-world environments is a separate issue. Therefore, research focused on enhancing model adaptability is crucial for improving performance in actual application settings.

The Claude Opus ARC-AGI test case vividly demonstrates the impact of harness on model performance. Initial model performance, which was only 30%, could be improved to 95% simply by applying harness as a wrapper and scaffolding. Furthermore, NVIDIA's AVO (Agentic Voice Over) achieved 100% performance.

These results prove that harness research is a powerful methodology that can dramatically boost the performance of existing models without fundamental weight modifications.

The AI agent swarm proposes an innovative process for automating research execution and paper writing. This system operates with multiple agents, such as purpose specification, scope definition, research, and validation agents, collaborating by dividing their roles. Users provide a few initial ideas (seed ideas), and the agents conduct their own research and derive results based on them.

Each agent performs specialized functions; for example, the purpose agent sets the overall direction of the research, and the scope definition agent defines the research's boundaries. Subsequently, the research agent utilizes high-performance computing resources like H100 nodes to conduct actual experiments, and the validation agent verifies the reliability of the results.

This multi-agent system demonstrates that it can generate tangible research outcomes and draft papers without human intervention.

The Voyager paper made a significant contribution to establishing the concept of 'skill' in AI agents. This research implemented a method where an agent performs specific tasks in a Minecraft environment, then extracts the acquired skills as system prompts for permanent learning.

This allows the agent to repeatedly utilize learned skills and combine existing skills to solve more complex problems when encountering new tasks. However, the question of ambiguity between skills and tool calls (APIs) still remains.

In a multi-agent system, several agents with different roles collaborate to improve outcomes. They are equipped with self-reflection mechanisms that help them improve their context, allowing them to operate smarter and more efficiently. Agents evaluate their work results through internal evaluators and go through a reflection phase to correct errors or improve performance.

This peer evaluation and feedback loop are key elements that enable the agent system to continuously learn and evolve, delivering more sophisticated results without human intervention.

Early forms of harness, namely Harness V1, had fundamental limitations. This system had a static structure that could not update or improve the system prompts or the harness itself. Since agent specifications and tool lists all relied on fixed settings, it was difficult to flexibly respond to changing environments or new requirements.

This meant that once configured, the harness could not evolve or optimize itself, which ultimately limited the full potential of the agents.

This meta harness provides the functionality for an agent to directly modify its own harness and, through this, transform into different forms of harness. In other words, its primary purpose is for the harness itself to generate better harnesses.

This approach enables agents to respond more flexibly to environmental changes and allows for an autonomous learning and improvement loop that continuously optimizes performance.

Test-time learning is emerging as an important direction that must be researched for the continuous performance improvement of AI models. This method utilizes Dagger-style online learning to directly update the weights of a Large Language Model (LLM) based on a small number of examples learned during model testing.

Unlike traditional static model learning methods, test-time learning allows models to integrate and adapt to new information in real-time, thereby maintaining high performance even in unpredictable or rapidly changing environments.

This approach is considered a key technology that helps models quickly adapt to new domains and continuously maintain an optimal state.

This strategy efficiently stores and accesses data by categorizing layers among model weights, active contexts, and external memory. This allows the model to find and process necessary information faster.

In particular, context compression strategies to reduce fine-tuning costs optimize the model's memory usage and contribute to increased learning efficiency. This is an essential element for maximizing LLM performance within limited resources.

Cache layer strategies play a crucial role in LLMs processing vast amounts of information and maintaining long-term conversations, ultimately enabling the construction of more intelligent and flexible AI systems.

YC's agent project began in January 2025 under the name 'Universal Agent.' Initially, it was a simple method, merely a system prompt in the form of a tool-use loop. However, by June 2025, the agent's capabilities greatly expanded as engineers increasingly utilized Claude Code and Codex.

Subsequently, the agent evolved to integrate with Slack and operate in a VM (virtual machine) environment, enhancing user accessibility and usability.

While these agents were clearly useful, users required significant setup to utilize them properly, and managing the swarm of systems itself was difficult. It was frequently necessary to SSH into individual VMs to resolve problems, much like a 'Whac-A-Mole' game.

Modifying individual agent instances also took considerable time, hindering the efficiency of the entire system.

To resolve the complexity of managing individual agents, YC shifted its architecture to a centralized system. All agent conversations and contexts were moved to a Postgres database for central management.

This brought about a paradigm shift, redefining the sandbox not as the agent's 'residence,' but as a 'resource' to be utilized when needed. By moving from individual machine management to centralized management, the system's overall context was exposed to the agents themselves, increasing efficiency.

To improve agents, YC introduced an automated improvement loop that applied a 'hill climbing' technique based on data. This is a process that helps agents learn and improve their performance autonomously.

However, the results of these attempts were somewhat mixed. A phenomenon known as 'protagonist syndrome' occurred, where agents focused only on their small part without seeing the entire system. This led agents to modify only specific parts without considering the overall system context, sometimes resulting in inefficiency.

Therefore, human intervention was confirmed to be essential before fully automating the system. Humans must complement the agents' 'protagonist syndrome' and provide guidance and feedback for optimizing the overall system.

This experience suggests the importance of finding a balance between automated improvement loops and intelligent human intervention in the development of AI agent systems.

This technique is similar to those used by leading companies like OpenAI and Anthropic to solve difficult problems. Through the grind tool, agents continuously attempt and improve until the set goal is reached, thereby increasing task completion rates and building a more robust system.

Another important issue in operating agent systems is that agents completely fail to understand social context. Humans have an intuitive judgment and mental framework for how much information can be shared and with whom, but implementing such abilities in agents requires significant effort.

This means that to prevent information leakage, the agent's knowledge scope must be finely linked with a permission system. YC already has a fine-grained permission setting system, which gives it an advantage in agent operation, but making agents learn social sensitivity and perform appropriate information sharing remains a complex challenge.