Stop Fine-Tuning to Fix Retrieval Issues

Files, databases, and APIs function as the primary repository for knowledge, requiring architects to decide how that data reaches the inference layer from the prompt or context.

Anant Srivastava emphasizes that knowledge primarily resides in various forms such as files, databases, and APIs. Retrieval, memory, and model adaptation are distinct tools, each suited for a specific function within the AI architecture.

Teams often escalate solutions for AI problems, moving from prompt adjustments to retrieval systems, and eventually to fine-tuning.

Srivastava notes that architectural decisions frequently arise from day-to-day product work over several months rather than from a unified design strategy. This results in an 'accumulated' architecture instead of a deliberately designed one.

One case involved an internal support assistant initially configured by adjusting prompts for tone.

As the system evolved, retrieval systems were implemented for refund policies, and product catalogs were directly inserted into prompts. The model was later fine-tuned using old support tickets.

This fine-tuning caused product names to 'leak' into the model weights. The system then generated incorrect outputs by citing outdated product information, persisting despite updated prompt configurations.

Anant Srivastava states that prompts should primarily define an agent's behavior, tone, and persona.

These use cases are characterized by being small, stable, and easily editable. Using prompts to store dynamic facts like product catalogs leads to increased token costs and the 'lost in the middle' phenomenon, where the model struggles to process extensive context.

Information that frequently changes or is too extensive for a prompt should be stored in memory.

Production AI systems require the ability to cite sources, which mandates the use of external memory. Agent memory manages user-specific interactions, while external memory handles broader datasets such as refund policies. Reasoning tasks depend on the model's inherent capability, not solely on the volume of provided context.

Knowledge requiring access control, such as user-specific information, must reside in memory to restrict visibility.

Effective RAG systems rely on proper chunking, metadata management, and filtering mechanisms. These help prevent 'rag mush,' a state where models are overwhelmed by irrelevant chunks of information.

Anant Srivastava advises against fine-tuning code assistants on a codebase because code changes too frequently.

Fine-tuning models should instead focus on learning 'reflexes' or behaviors, not transient facts. Attempting to embed an entire codebase into a prompt is both inefficient and ineffective for dynamic environments.

The rate of data change serves as the primary metric for deciding whether to bake information into weights, as model weights are intended only for information that has stopped changing.

The rate of change is the primary determinant for deciding whether to integrate data into model weights. Many teams mistakenly fine-tune models on dynamic documentation or runbooks to address retrieval problems.

This practice results in the model providing outdated information, as fine-tuning on a knowledge gap merely bakes in a snapshot of data that may quickly become obsolete.

Anant Srivastava advises that if a model is failing to provide correct answers, the initial focus should be on resolving retrieval problems.

Fine-tuning should not be the immediate solution for addressing gaps in model performance.

When humans consistently correct recommendations from a frontier model, a stable 'center' of patterns emerges. Fine-tuning should target this stable core, while continuous monitoring for drift is essential to prevent performance degradation as patterns evolve.

Anant Srivastava argues that static knowledge, such as medical claims coding (e.g., ICD-10 codes), should reside in memory, not model weights.

Fine-tuning should focus on teaching the model 'reflexes,' like understanding input formats and selecting the correct format. Models should not be fine-tuned on information that is prone to change or requires extensive training data.

The main reasons for fine-tuning should be to enhance model capabilities or to optimize costs by leveraging smaller models.

Over time, successful retrieval patterns can be moved from memory into model weights through fine-tuning, reducing the need for constant contextual examples.

For instance, fine-tuning for specific formats, such as medical notes, can streamline processing and improve efficiency.

The AI model itself is only a small part of a larger system. Anant Srivastava emphasizes that the success of an AI system depends significantly on the architecture built around the model.

This 'harness' is crucial for storing, retrieving, and circulating information effectively. A well-designed system enables an agent to continuously improve its performance through an iterative loop of task execution and learning.