Beyond RAG: Building Relational Context Engines

Peter Werry defined Retrieval-Augmented Generation (RAG) primarily as semantic similarity search or vector retrieval, representing only half of the broader retrieval process.

The objective of the workshop was to develop the other half: a system capable of transforming natural language into executable database queries, which was demonstrated through six progressive pull requests.

Engineering teams typically scatter contextual information across various tools, including pull requests, issue trackers like Linear and Jira, wikis such as Notion and Confluence, and chat platforms like Slack and Teams.

While this data can be semantically queried, it inherently contains strict structural metadata, including authors, creation dates, and labels, which traditional vector dumping into vector stores often fails to leverage for specific inquiries.

RAG effectively supports content-based discovery, such as locating general documents or broad pull request discussions.

However, it falls short when queries involve temporal or relational logic, requiring precise filters or relationships between data points.

For instance, a query like 'PRs I merged last week' necessitates filtering by specific authors, merge states, and timestamps, which a standard vector similarity search might fail to address by returning semantically related but outdated pull requests.

Bridging the gap between raw semantic search and precise data retrieval, the AI model generates a formal query rather than answering the user directly.

This approach bridges the gap between raw semantic search results and the precise, structured data retrieval required for specific inquiries, with reliable query generation being a fundamental requirement for advanced context engines.

Before generating a query, the Large Language Model (LLM) must understand the underlying database schema and map natural language entities, such as user names like 'Peter,' to corresponding database identifiers like GitHub usernames.

To ensure security, the system must prevent the execution of untrusted queries, such as malicious JavaScript, and incorporate a self-correction mechanism to handle initial query failures gracefully.

The engine dynamically reconstructs the schema since it does not rely on a predefined structure, achieved by sampling 300 documents to infer their structural relationships.

Documents are collapsed into a single tree while null fields are skipped, and procedural code is favored over LLM-based extraction for structure validation due to the non-deterministic nature of LLMs.

This dynamic approach allows the system to adapt to schema-less data sources effectively.

Identity mapping becomes crucial when natural language inputs, such as a user's name like 'Peter,' do not directly correspond to database identifiers, such as a GitHub login.

These mappings, though seemingly tied to the schema, can be determined dynamically by the LLM, with initial implementations leveraging simple fuzzy matching techniques like Levenshtein distance for efficiency.

By passing the schema and specific tools to an LLM, the system achieves temporal accuracy through injected date and time information.

To ensure temporal accuracy, the current date and time are included in the LLM prompt, enabling the model to accurately map natural language queries like 'open PRs by Peter' to precise identifiers such as GitHub user logins.

Uncontrolled output from LLMs poses security risks, including prompt injection through JavaScript execution or unauthorized function calls.

The system mitigates these risks by implementing a 'deny by default' strategy, utilizing strict allow-lists for all operators and fields involved in query execution.

Furthermore, metadata fields critical for tenant isolation are hidden from the LLM, and any accidental leakage of such information in the output is promptly stripped to maintain data integrity and security.

Query plans can fail due to various issues, including validation errors or execution problems such as illegal inclusion or exclusion patterns.

The system incorporates a feedback loop that passes error messages back to the LLM, allowing it to refine and correct the query, with maximum attempt limits in place to prevent infinite loops during this self-correction process.

Integrating full-text search (FTS) significantly extends the model's capacity to handle more complex user requests.

This involves creating a full-text index and combining it with existing filters like author and state within the same match operation, enabling the engine to move beyond exact matches and process more sophisticated inquiries.

Traditional RAG struggles with questions that involve specific states or temporality, often failing to retrieve the correct data.

The relational context engine supports advanced functionalities such as filtering, aggregations, time-scoping, and full-text search.

Key reusable concepts for this system include schema-aware prompting, dynamic schema discovery, and robust identity resolution, with the error-handling loop identified as essential for overall system reliability.

To scale schema management in production, an additional LLM step is introduced to summarize schema collections with concise descriptions.

LLMs are now highly capable of identifying and loading only the schema relevant to a specific query, thereby reducing context size.

Data ingestion scaling is approached as a standard distributed systems problem, typically managed using webhooks.

Slack captures crucial conversational decision-making and past context, making it a valuable source for understanding the 'why' behind actions.

Pull request comments are another significant repository of organizational history, often referred to as a 'black hole,' which Unblocked leverages to create 'best practice memories' by distilling past discussions.

By bridging vector-based conceptual matching with keyword-focused retrieval, agents synthesize comprehensive query strategies that neither approach achieves alone.

Combining these two approaches allows an agent to use vector search as an initial step to gather broad context, subsequently forming more specific queries based on keyword identification.

Documents often form JSON-like structures that can be merged to extract specific tokens, which are then analyzed to distill type information.

The current algorithm employs heuristic-based rules, such as character count constraints (e.g., 2 to 8 characters), and identifies enumerations by detecting recurring patterns within the data.

Analyzing failed searches reported by customers helped refine schema heuristics, as Unblocked employs static schemas in production for reliability.

Schema heuristics were refined through extensive empirical trial and error, leveraging insights from customer-reported failed searches to improve accuracy.

Schema discovery occurs once at the initial stage, with subsequent steps focusing on extracting entity names.

If an agent fails to invoke an extraction tool, a forced user message is injected to prompt the agent, and validators continuously monitor agent actions to ensure that all mandatory query steps are executed correctly.