Spotify Transforms Recs with LLM-Native AI

Raimond, with a background in personalization from Google Search and Netflix, highlighted the ambition to "turn the recommendation system on its head" by deeply integrating Large Language Model capabilities.

The content catalog is equally massive, encompassing over 100 million music tracks, along with a wide array of podcasts, audiobooks, and videos, presenting an immense challenge for personalized content matching.

Spotify's personalization journey began with manual curation, evidenced by over 10 billion user-created playlists on the platform.

The second phase introduced recommendation engines, exemplified by the launch of Discover Weekly in 2014, which focused on algorithmically matching users to content.

The current phase, termed generative personalization, moves beyond simple matching to dynamically create interactive experiences, shifting the system's focus from 'guessing' through ranking to 'reasoning' and from 'black-box' algorithms to 'steerable' ones.

Spotify DJ offers users the ability to steer their musical direction in real-time by tapping a button, allowing the algorithm to adjust the session based on immediate feedback.

Prompted Playlists enable users to request highly specific content, ranging from general themes like 'bands playing in San Francisco tonight' to functional requests such as 'music for a run.'

This dynamic system can adapt to user input, modifying playlists on the fly, for instance, by changing music based on different phases of a workout.

Spotify launched a 'taste profile' feature in New Zealand, allowing users to understand and introspect what the algorithm has learned about their preferences in natural language.

This feature empowers users to directly edit and refine their profiles, correcting algorithmic assumptions or adding missing information, such as excluding kids' Disney music from personal recommendations if a shared device has influenced the profile.

Users gain explicit control over their recommendations, allowing them to signal "not my taste please" to prevent certain content from being recommended in inappropriate contexts.

Beyond direct filtering, these profiles support aspirational goals, such as learning new languages or exploring new genres, ensuring that the system's recommendations align with the user's current intent.

This new capability includes generating daily audio briefs that are tailored to the user's community and specific interests, with content produced on a daily basis.

Spotify's Large Taste Model (LTM) consolidates all recommendation applications into a single, unified architecture, combining prediction with reasoning to generate real-time, user-promptable experiences.

This system understands historical interactions and has been adopted by one in four US Premium subscribers daily, yielding significant performance gains in areas like autoplay, podcast discovery, and DJ interactions.

These models are fine-tuned to understand both natural language requests and these specific semantic tokens, allowing for richer interactions; for example, a user asking for a 'podcast on morality' could receive a specific episode along with a natural language explanation.

Jacqueline Wood described Spotify's NEO training paradigm, a four-stage process designed to ground LLMs in specific domains and induce new capabilities without forgetting existing language abilities.

The stages include: semantic foundation (creating ID tokens), domain grounding (aligning ID tokens with language embeddings while freezing the LLM backbone), capability induction (multitask instruction tuning), and optional post-training (e.g., RL fine-tuning).

Spotify's research showed that multitask models consistently matched or outperformed single-task models across various recommendation tasks, indicating positive cross-learning.

This approach proved particularly effective for 'cold start' entities, such as improving audiobook recommendations by leveraging existing data from podcast consumption.

Ablation studies revealed that using a randomly initialized backbone, instead of a pre-trained open-weight LLM, resulted in the most significant degradation of model performance.

The studies confirmed that a frozen backbone during domain grounding effectively preserves natural language and world knowledge, and these findings hold true across both Qwen and Llama models.

Comparing inference strategies, beam search, even without constrained decoding, achieved 98% validity in semantic IDs, significantly outperforming top P sampling in accuracy.

Although beam search introduces more latency, Spotify determined that this trade-off was acceptable given its superior accuracy, particularly when constrained decoding is used to target specific content types like new releases.

Deployed in production, NEO is already powering podcast discovery and has successfully shifted user behavior toward listening to a wider array of unfamiliar content.

Jacqueline Wood noted that traditional offline metrics are insufficient for evaluating generative and explanatory recommendations, requiring investment in grounding LLM judges with meaningful data for reliable evaluation.

By using textual user profiles to summarize listening history, Spotify achieved 75% alignment with human preferences, further boosting alignment by 5% overall (and 91% on ambiguous queries) by incorporating behavioral signals into search queries.

Traditional offline evaluation metrics fall short in assessing generative and explanatory recommendations, as they cannot measure whether recommendations make sense to a user or if explanations are accurate.

These metrics also fail to determine if recommendations truly align with user intent, necessitating the introduction of LLM judges as a superior alternative for qualitative evaluation.

Cranfield-style evaluation collections, which involve pooling candidates from multiple sources for ranking, are typically resource-intensive due to the need for manual human assessment.

Spotify successfully trained an LLM judge to replicate human system rankings, achieving an agreement value of 0.87 with human assessors, thereby scaling up these evaluations efficiently.

Jacqueline Wood emphasized that the industry is now entering a new era of generative personalization, moving beyond static recommendations.

Spotify is utilizing open-weight LLMs to facilitate language-steerable interactions, aiming to provide users with more dynamic and controllable personalized content experiences.