Just-in-Time Ontologies Reshape Data Strategy

Hrishi Olickel questions whether the industry spent a decade chasing distributed architectures based on the outdated premise that compute and storage were inherently slow and expensive.

Traditional data transformation was historically treated as a costly operation, a perspective now challenged by the availability of affordable, accessible intelligence.

By integrating a new model and utilizing Just-in-Time (JIT) transformations, the entire task was completed within 10 days for less than $800, demonstrating the efficiency of the new approach.

The speaker contends that the failure of ontologies extends beyond mere organizational buy-in, rooted in the fundamental flaw of seeking a singular 'super schema' that can universally address all enterprise needs.

There is no universal concept of a customer, as its definition is intrinsically linked to the specific question being asked, such as for billing, support, or compliance purposes.

Effective data management requires satisfying four distinct schemas: the source, storage constraints, the decision being made, and any prior assumptions or models.

JIT ontology involves performing data merges and transformations only when the specific decision space is known, rather than attempting to define all schema requirements prematurely.

This approach facilitates the necessary discernment to merge recorded context with the context required for a particular decision, making information useful precisely when needed.

Most data engineering efforts fall into three operational categories, according to Olickel, who emphasizes the importance of embedding intelligence into pipelines that scale and mature independently.

The regimes include 'write once, run once' for single-use data mining; 'write many, run once' for large-scale migrations; and 'write many, run many' for repeatable, core data pipelines.

Modern systems possess the capability to self-optimize, selecting models, conducting pilots, and scaling independently once given a problem.

These systems primarily require a defined problem, a safe failure pathway, and a designated workspace to autonomously discover effective solutions.

A historical data failure involved conducting AB testing with an insufficient sample size, such as only 50 users, leading to unreliable results.

The modern risk involves interactive AI agents that consistently provide quick answers, which can result in 'prod nightmares' where statistically invalid numbers are presented as factual, arguing for an erroneous outcome.

Beyond pipeline design, Olickel notes that intelligence serves as an engine for executing specific tasks and as a catalyst for models that self-optimize through iterative refinement.

The first role involves building and designing the pipeline, which includes problem breakdown, specification, and cost authorization, demanding 'slow, heavy' models capable of deep thought.

The second role is focused on executing tasks within the pipeline, such as reading PDFs or interacting with databases, requiring 'fast, specialized' models; the third category involves 'hill climbers' or 'surfers' models that initiate a process and then self-optimize.

Southbridge AI tested three design architectures on a small development set for entity resolution in Ohio campaign finance data before scaling the solution.

Cheap reasoning models performed comparably to more advanced models, but at a mere 2% of the cost, demonstrating significant efficiency.

Refining the logic for household definition dramatically improved performance from zero to 13 successful 'yes' results, and the system ultimately processed 23,317 cases with explicit labels for evidence and audit trails.

This achievement resulted in an estimated cost reduction from a projected $10,000 to near zero for similar future state projects, as an autonomous agent (4A) ran for 10 days, testing over a dozen architectures, performing over 4,000 tasks, and processing roughly 6 billion tokens.

Larger models can paradoxically be less effective in certain data schema assumptions, often clinging to their inherent 'priors' or beliefs about data formatting with greater vehemence.

One model, for example, misclassified 15,658 in-kind contributions due to inconsistencies in naming conventions, while another stripped negative signs from loan amounts, erroneously transforming $784,000 in net debt into $5.3 million of debt, an error validated by multiple models.

Such validated, high-confidence errors are emerging as the most significant class of issues in contemporary data work, highlighting the challenges of model assumptions.

Models originating from different families and pedigrees often exhibit greater disagreement and possess distinct priors compared to models within the same group.

Errors frequently cluster within instances where diverse models disagree, while identical model families may inadvertently produce the same incorrect results due to shared, erroneous sign conventions.

The first pillar is a stable deployment surface, which functions as a harness providing consistent logs, checkpointing, and budget enforcement for agentic systems.

The second pillar emphasizes bounded experiments, allowing for control over costs, time, and resource access during developmental phases; the third is comprehensive know-how and methodology to apply learned patterns and debugging techniques effectively.

The fourth and most critical pillar is a failure surface, which defines a constrained space or set of rules against which a system can iterate to solve problems and learn safely.

The JIT ontology process involves six stages: first, specifying and splitting the job into smaller, typed cases, such as breaking down 200,000 donor records into specialized cases for entity resolution.

Second, defining a failure surface using manual baselines or automated schema checks; third, running minimal bounded pilots to test the architecture; and fourth, moving successful checks into reusable 'codons' (code segments).

Fifth, deploying working pipelines to production, where they can slowly mature, and sixth, continuously refining and maturing the system by swapping for cheaper, faster models while simultaneously tightening requirements.

Olickel asserts that there is no universal ontology, nor was there ever one; instead, ontologies serve as intermediaries between intelligence and desired results, valid only when their context and supporting evidence are fully preserved.

This approach renders any schema valid, provided its context and evidence are maintained, enabling information of any format to become useful within its specific application.