Legora Scales AI Legal Search with Turbopuffer

Its core functionalities encompass contract review, document drafting, and extensive legal research, acting as a centralized workspace that streamlines various legal workflows.

Jacob stated, "We're a collaborative AI platform for legal work. And so that means we have law firms that are clients and we have uh in-house legal teams that are clients."

Legora differentiates between two primary types of search: project search and legal research. Project search focuses on analyzing specific documents relevant to a single task or project, with project sizes ranging from dozens to millions of documents. In contrast, legal research involves a broader exploration of laws, cases, and regulations across vast datasets.

Jacob explained, "So at Lor there's two types of search that we do. There is project search and legal research."

This monolithic setup meant that all client tenants shared a single blob storage and search instance. While simple to implement at first, this approach presented limitations as the company expanded.

As Legora grew, it faced strict geographic data residency requirements, necessitating data processing within specific regions such as the US, EU, and Australia. This compliance demand forced the company to multiply its entire infrastructure stack across different regions, significantly increasing operational overhead due to the management of multiple ElasticSearch clusters.

Jacob noted, "And so we had to basically um well here's a scaling graph. We had to move to multiple elastic searches."

Enterprise clients imposed rigorous security standards, primarily demanding full physical isolation for all their data. Legora addressed this by implementing customer-managed encryption keys for data at rest, allowing clients to retain control by revoking access to their encryption keys as needed. This ensures a high level of data security and client autonomy.

Jacob stated, "Number one they have is they'll ask for full physical isolation of all of their data."

Legora transitioned its core search infrastructure from ElasticSearch to Postgres to simplify its systems and leverage existing OOLTP workload setups. This strategic move involved using PGvector for vector embeddings and disk Approximate Nearest Neighbor (ANN) search, alongside TSvector for full-text search functionality. This consolidation aimed to streamline operations and enhance system manageability.

Jacob mentioned, "So we moved from elastic search to Postgress and I imagine a bunch of you guys are like why would you ever put your vectors into Postgress."

NVMe SSD cache is treated as volatile memory, bypassing standard disk encryption rules, and disabling disk cache for specific workloads helps maintain high performance with memory-only caching.

Simon Eskildsen said, "All of the namespaces needed to be physically at rest with different keys and as separated as possible."

P99 latencies showed even more substantial gains, which is crucial for agent-based workflows that execute numerous queries. This cumulative latency reduction is critical for maintaining responsiveness in complex legal AI applications.

Jacob Lauritzen stated, "Latency is improved in order magnitude basically. And these are median latencies. So, P99 were even better."

Legal data is structured with strict hierarchies—city, county, state, and federal laws—and this structure must be preserved in search. Furthermore, temporal validity and regulatory exemptions introduce additional layers of search complexity, requiring queries to involve heavy fan-out due to complex graph-like relationships within the law.

Jacob Lauritzen explained, "It's hierarchical. You know, you have cities and you have counties and you have states and you have federal law and it's the same all around the world."

Elasticsearch proved cost-prohibitive for Legora's large-scale legal datasets, prompting the transition to Turbopuffer. Turbopuffer's architecture allows different jurisdictions to be managed as separate namespaces, enabling a tiered storage strategy. This means high-traffic data, such as EU law, can be kept in "hot" storage for quick access, while lower-traffic data, like Danish law, can reside in "cold" storage. This design also supports a high tolerance for latency during retrieval from cold blob storage, optimizing overall cost efficiency.

Jacob Lauritzen mentioned, "Elastic search just got extremely expensive because we have to have everything there but um with turbopuffer we can basically take different jurisdictions and we can make them name spaces."

Turbopuffer's design, metaphorically described as 'puffing' data, focuses on mastering when data should reside in specific memory hierarchies for efficient query execution. This means frequently queried data, such as EU law, is strategically positioned closer to NVMe and memory levels. The database architecture optimizes round-trips and data access patterns by considering the data's location within the hierarchy, with the ultimate goal of pushing as much data as possible into optimal storage tiers.

Simon Eskildsen elaborated, "One of the other explanations of the name of Turbopuffer is that it's about puffing into the different memory hierarchies and really mastering when data should be in particular memory hierarchies."

Turbopuffer organizes vectors into clusters, treating them as points in a massive coordinate system, which contrasts with traditional graph-based approaches that suffer from latency penalties when navigating nodes on disk or object storage. This tree-like, cluster-based organization is designed to balance vector search performance with different memory tiers. By avoiding graph navigation issues inherent with disk-based storage, Turbopuffer optimizes data retrieval across various memory hierarchies.

To enhance search latency, Turbopuffer optimizes tree levels by keeping root centroids in DRAM, which are accessed during every search to identify relevant clusters. Leaf nodes, containing the actual data, reside on SSDs. This tiered approach minimizes the amount of data loaded into DRAM, making it cost-efficient and suitable for massive-scale datasets like web-wide indices, by focusing on accessing only the necessary upper levels of the tree.

Simon Eskildsen explained, "Now the root centroidids further up the tree you can imagine are part of every single time you search right there. We're always trying to figure out which clusters that we're in and we're always looking at the upper levels of the tree."

Full-text search in Turbopuffer functions essentially like a hashmap, where each token from a document is mapped to a set of document IDs. For multi-term queries, the system performs an intersection of these sets to find matching documents. BM25 scoring is then applied to account for word rarity, minimizing round trips and memory bandwidth requirements to ensure efficient and relevant search results.

Simon Eskildsen described, "The way the text search works is essentially you can think of it as a hashmap. You have a big document and then you take every single one of the tokens and you put them into the key in the hashmap."

Its performance heavily relies on efficient index compression and minimizing bandwidth usage. This complexity stems from the need to process and retrieve relevant textual information across massive, dynamic datasets with high precision, making it a significant engineering challenge.

Simon Eskildsen observed, "And counterintuitively to most people, text search at web scale is more difficult and more computationally expensive than doing vector search."

With 70 to 200-plus tenants, avoiding separate ElasticSearch databases for each client is crucial for scalability and cost-efficiency. This shift allows Legora to redirect its focus from maintaining complex infrastructure to accelerating product development, driven by data residency requirements and overall cost optimization.

Jacob Lauritzen highlighted, "We have 70 plus tenants. We have 100, we have 200 tenants. You know, if we had to have separate elastic search databases for each of these, it would be just hell."