LLMs Reshape Ranking Systems, Beyond Scoring

For instance, a conference traveler, a family traveler, and a backpacker would each have distinct preferences for a hotel booking, requiring the ranking system to adapt to these varied needs. This approach necessitates balancing competing objectives such as price, diversity, and specific business goals, making the task of ranking highly dynamic and context-sensitive.

Goel emphasized that "Ranking has never been about finding the single best item. It's about finding the single the best ordering for a particular user and context."

Traditional ranking architectures involve a standard process of candidate retrieval, signal prediction, and re-ranking, but face significant limitations when mapping complex human intent to objective functions. Engineers often struggle to translate nuanced business requirements into precise mathematical equations with specific thresholds and optimization weights. This translation process is where difficulties arise, as it relies heavily on offline experiments, parameter tuning, and extensive A/B testing to refine performance.

Goel noted the challenge: "Now every part of this sentence sounds perfectly reasonable but somewhere now an engineer has to translate this into a mathematical objective function with different thresholds and optimization weights and that's where things become difficult."

This capability allows engineers to simply describe desired outcomes in natural language, directly leveraging the model's understanding. By shifting the engineering focus from hand-crafted weights and equations to language-based instructions, LLMs simplify the process of capturing nuanced preferences.

Goel outlined the potential: "What if instead of translating human intent into objective functions, we could just optimize for language directly. Now this is what LLMs make possible. So we just simply describe what we want."

While LLMs offer advantages in direct language optimization, they introduce new engineering trade-offs, particularly regarding reliability and evaluation. Key challenges include ensuring that language-based objectives can be optimized consistently and precisely, especially when dealing with vague terms like 'unless quality drops significantly.' Determining how to quantify such imprecise instructions and verifying the consistency of model reasoning are critical for trusting these decisions and developing robust evaluation frameworks.

Goel raised pertinent questions: "Can language be optimized reliably? How precise is unless quality drops significantly? Or can models reason consistently? Can we trust this these decisions? And how should we evaluate them?"

This framework addresses whether prompting alone is sufficient for a general-purpose LLM or if domain-specific training is necessary. It also covers how to refine model behavior to capture more nuanced preferences, ensure performance and latency meet production environment requirements, and establish effective validation metrics for success.

Goel summarized the framework: "There are really five things to ask. Uh if we have a general purpose LLM is prompting alone enough if not how do we teach the model about our domain? And once it understands our domain, how do we refine its behavior to capture more nuanced preferences?"

When provided with a straightforward instruction, such as acting as a "hotel ranking assistant" for an anniversary trip, the model can rank hotels and explain its reasoning. This capability highlights the inherent understanding of these models without requiring extensive additional training for basic ranking tasks.

Goel gave an example: "We start with the general purpose LLM. We give it a simple prompt. Hey, you're a ranking uh hotel ranking assistant. rank these hotels for an anniversary trip to Napa and explain your reasoning."

In contrast, LLMs reason from natural language intent, offering a different strength by interpreting complex textual queries and contexts.

Combining both approaches is a standard practice for modern ranking systems, leveraging the strengths of each. This hybrid strategy allows systems to benefit from the statistical power of behavioral models for predicting user actions and the semantic understanding of LLMs for interpreting nuanced intent.

Goel articulated the difference: "Behavioral models learn from millions of historic interactions. They're really good at predicting what the users are likely to do. ... But LLMs excel in a different way. They reason from language."

Historical logs do not always account for subtle judgments or the evolving understanding of what is "better" in a given situation. For example, when asked for a preference for a query, humans are more likely to select a boutique resort over a popular chain if the context suggests a unique experience.

Sometimes models need to learn what constitutes a superior choice rather than merely repeating past selections, which may carry inherent biases. Preference data, derived from human judgments, can capture these subtle distinctions that historical interactions often miss, allowing the model to incorporate context beyond average historical behavior.

Goel illustrated this point: "if you ask a human for this query what they would prefer, they're more likely to say the boutique minyard resort because they have incorporated context that goes beyond the average historical behavior."

RLHF typically involves a two-step process: first training a reward model, which acts as the learned objective function, and then using reinforcement learning to fine-tune the language model based on this reward. This method can be computationally intensive and complex to implement.

DPO streamlines this by directly training the language model using pairwise human preferences, which means it learns directly from examples of preferred and dispreferred outputs. This simplification allows DPO to achieve comparable quality to RLHF but with significantly less complexity in its training pipeline. By removing the intermediate reward model, DPO makes the optimization process more efficient and easier to manage.

Goel explained the benefit: "instead of explicitly learning a reward model and then running reinforcement learning DPO just optimizes the language model directly from the preference pairs."

While individual examples might focus on identifying the "best hotel" for a user and query, real-world systems deal with diverse goals like affordability, showcasing new inventory, and promoting sustainable options. Further complexity is added by business constraints, such as revenue protection, which must also be integrated into the ranking logic. Balancing these varied and sometimes conflicting requirements introduces significant ambiguity and intricate challenges into the overall ranking task.

Goel highlighted the distinction: "Every example that we've looked at has optimized a single objective, which is identify the best hotel for this user and query. But that's not how production systems work."

Hard constraints, which are non-negotiable, are enforced deterministically either before or after the LLM ranking process, preventing the LLM from attempting to satisfy them through reasoning. This ensures essential rules are always followed, regardless of the model's output.

Conversely, soft constraints leverage the LLM's reasoning capabilities to maintain flexibility, allowing the model to consider and optimize for these preferences within its ranking decisions. Human preference alignment, often achieved through Direct Preference Optimization (DPO), plays a key role in ensuring consistent reasoning across different requests for soft constraints. This hybrid approach allows the system to balance strict rules with adaptable, intelligent decision-making.

This process involves the teacher model generating both rankings and explanations for a comprehensive training dataset. By leveraging the teacher's advanced capabilities, the smaller student models can learn complex reasoning patterns and nuanced preferences without needing to be as computationally intensive.

The entire distillation process is conducted offline, which is critical for real-world deployment. This offline training enables the deployment of leaner, more efficient student models into production environments. These smaller models can then operate with lower latency and reduced computational cost while still delivering ranking quality comparable to their larger counterparts, making them practical for large-scale applications.

Goel noted the process similarity: "It should again feel familiar because once again, we're just doing supervised fine-tuning. The difference is that instead of humans creating the labels, we can ask a much larger and more capable teacher model to generate them."

Not all queries require the same level of high-level reasoning; some are incredibly simple and can be handled by less complex models. Routing mechanisms analyze query difficulty and direct it to the most suitable model, preventing over-utilization of powerful, resource-intensive LLMs for straightforward tasks. Cascading involves escalating a query to more complex models only when initial outputs from simpler models are insufficient.

Goel explained the need: "Some queries are incredibly simple. Others require more deeper reasoning. So that's where uh routing and cascading come in."

The first tension, prediction versus reasoning, involves balancing historical behavior signals with semantic intent to predict user actions while understanding underlying meaning. Precision versus expressiveness refers to managing natural language objectives alongside structured production constraints to achieve both flexibility and accuracy.

The third tension, generality versus controllability, addresses aligning foundational LLM knowledge with platform-specific policies to ensure both broad applicability and precise adherence to rules. Finally, quality versus cost focuses on optimizing ranking accuracy while managing constraints like latency and infrastructure budget. These tensions form a framework for navigating the complexities of integrating LLMs into robust, scalable ranking systems.

Goel outlined these tensions: "I think of these as four engineering tensions. ... And the first one is prediction versus reasoning. ... The next is precision versus expressiveness. ... The third is generality versus controllability. ... And the final is quality versus cost."

Traditional models bring decades of operational experience, robustness, and computational efficiency to the table, excelling at large-scale data processing and established patterns. LLMs, on the other hand, contribute deep semantic understanding, sophisticated intent interpretation, and optimization aligned with human preferences.

The key question is how to integrate these strengths effectively, rather than debating which approach will prevail. By leveraging the statistical power of traditional methods with the nuanced understanding of LLMs, ranking systems can achieve unprecedented levels of accuracy and user satisfaction. This hybrid approach promises to unlock new capabilities that neither model could achieve alone.

Goel reframed the debate: "We often ask whether LLMs will replace traditional ranking models and after working through this entire life cycle I don't think that's the right question. I think the more interesting question is how do we combine the strengths of both"

For example, a user searching for a "wedding guest dress in Hawaii" requires the system to process not only the dress type but also contextual information like weather data and social etiquette (e.g., avoiding white at a wedding). Such queries cannot be resolved by a single model component.

Instead, these systems involve multiple specialized components that interact: one might query a database for relevant products, another might access real-time weather information, and yet another performs the core reasoning to combine these insights. This modular approach allows for intricate problem-solving, where different parts of the system contribute to a comprehensive, context-aware ranking. Goel noted, "you need a bunch of these things that need to be answered and these could be different, say um there could be something that's talking to the database, something that is actually doing the reasoning. So there could be like multiple components."

This method involves collecting pairwise human preferences, where annotators compare two outputs and indicate which is superior. These preferences are then used to train a "judge LLM" that mimics human annotators, effectively learning complex rubrics without explicit rule-coding.

By leveraging DPO, the model can internalize subtle distinctions and complex criteria for what constitutes a "better" or more aligned output. This allows for a deeper understanding of user needs and contextual appropriateness than what can be achieved with simple instructions. The process trains an LLM to become an expert in human taste and preference, making it a powerful tool for refining ranking systems.

Goel elaborated on DPO: "the best way that the industry has been solving for this is direct preference optimization. So we actually get uh pair wise preferences from humans. we train an another LLM model to mimic the human annotator to become that judge"

This dual-track enhancement involves scheduled retraining and updates for traditional models, ensuring they remain robust and efficient. Simultaneously, LLM components undergo fine-tuning and refinement to improve their semantic understanding and preference alignment.

An emerging area of continuous improvement is automated prompt refinement, where agents evaluate Customer Experience (CX) to automatically optimize prompts for better LLM performance. This iterative process allows both parts of the hybrid system to adapt to new data, user behaviors, and evolving business requirements. This ensures that the ranking system remains effective and up-to-date in a dynamic environment.

Goel noted the prevalence of hybrid systems: "most systems that I have seen in practice or read about online, they're still hybrid. They still have the traditional ranker plus some LLM reranker component. And so anything that the continuous improvement basically continues on both sides."