Harness Engineering: Beyond Basic AI Prompting

The conceptual design of skills as extensions of existing coding harnesses provides a more robust and scalable method for creating reliable AI solutions, addressing the limitations of simple, conversational prompts.

Trends like 'purple gradients' rapidly shift to new clichés, such as 'cloud beige,' illustrating how attempts to explicitly forbid certain aesthetics often push models to adopt the next most common pattern within their latent space.

Prose-based prompts struggle to overcome the inherent bias of AI models, often referred to as the model's internal 'gravity.' This default tendency resists even extensive and well-crafted natural language instructions.

Even 250 lines of carefully written prose are insufficient to override default model tendencies, with directives against specific fonts or colors merely shifting the model to the next predictable pattern rather than achieving true design control.

AI models frequently fail to objectively evaluate their own work, often rating their creations very highly due to an anchoring bias on initial outputs, akin to grading one's own homework.

Adversarial prompting addresses this by employing two sub-agents that operate independently, without access to each other's work, ensuring unbiased critique.

This method prevents the model from overreacting to minor errors or overlooking significant flaws by combining deterministic linting with a blind agent evaluation process.

The separation of evaluation tasks provides a more accurate and critical assessment of the AI's generated content.

These automated systems ensure continuous adherence to guidelines, enabling personalization through custom ESLint rules and syntax guidelines, with essential feedback loops keeping models on track.

Automated design hooks, while beneficial, frequently trigger false positives, necessitating a robust mechanism for creating ignore rules to prevent user frustration.

Tools like Impeccable address this by offering file-level and CSS-rule-level ignore capabilities, providing granular control over automated feedback.

Such granularity is crucial for preventing AI design tools from becoming annoying to users and ensuring the system remains helpful rather than disruptive.

Platforms like Claude, Cursor, and Codex have distinct methods for spawning sub-agents, with tools such as 'ask user' being restricted to plan mode in Codex, unlike in Claude.

Developers must write conditional code to manage platform-specific limitations effectively, ensuring broad compatibility and functionality.

Cursor and Codex, for instance, spawn foreground tasks that block the actual chat thread, leading to different operational dynamics compared to true background processing, while tail watch functionalities are throttled differently depending on the tool.

Each AI model exhibits distinct overfitting tendencies, such as Gemini's propensity to excessively animate pictures, requiring explicit instructions to avoid unwanted effects like hover animations.

Codex, for example, shows biases for poor letter spacing and rounded or hairline borders, necessitating ablation testing to identify and mitigate these model-specific instruction biases.

Achieving consistent instructions across multiple AI models, such as Claude and Gemini, requires model-specific builds to prevent overfitting, as a rule for one model can have an inverse effect on another.

XML blocks are used to inject specific avoidance rules for each model, ensuring that instruction compression by models is managed through rigid gate structures to maintain quality.

To ensure end-to-end task completion, instructions must be made 'unskippable' by forcing models to log every gate result.

This approach prevents models from wiggling out of difficult situations and ensures disciplined adherence to complex instructions.

While parts of a skill may remain cached, the dynamic call-out portion does not, as the nature of shell execution dictates that results are treated as always subject to change.

A custom internal Eval harness meticulously recreates specific tool conditions for various models, guaranteeing that instructions are consistently followed across diverse AI environments through rigorous testing.

Paul Bakaus has developed an LLM that acts as a user against another LLM, incorporating a 'mixture of expert design judge' to evaluate AI skill outputs across various niches and models.

Ablation testing systematically removes specific rules from the source code to confirm their functional impact, ensuring each rule's necessity and effectiveness.

Deterministic detection engines verify the changes made by LLMs against a predefined set of rules, solidifying a transition from 'vibes-based' development to a battle-tested pipeline.

This rigorous testing framework, including 20 different niches and models like GPT-4, GPT-3.5, Opus, and Sonnet, ensures high-quality and reliable AI skill performance.

Bakaus argues that individual users should be empowered to write and package their own AI skills, rather than relying on unvetted public offerings, to ensure quality and reliability.

The current ecosystem suffers from a significant lack of standardized distribution methods and rigorous battle-testing, leading to skills that often fail when run on models not initially tested by their authors.

This issue highlights a broader problem, as there is currently no industry standard for skill distribution across the multitude of different AI harnesses available.