Uber Eats Details Multimodal AI Agent Deployment Strategy for Image Optimization

Uber Eats operates at a substantial scale, reporting a $90 billion annual run rate and a 20% year-on-year growth. The platform is active in 10,000 cities globally, connecting millions of restaurants with customers. This extensive reach underscores the impact and complexity of deploying AI solutions across its diverse marketplace.

Image quality is a critical factor for delivery platforms like Uber Eats because photos serve as the initial point of contact for customer impressions. High-quality images directly influence click-through rates and cart conversion rates, making them vital for driving sales. Furthermore, the importance of video content is also on the rise, indicating a broader trend toward richer visual engagement.

Small merchants often face significant challenges in producing high-quality food photography due to a lack of time, know-how, and financial resources. This issue becomes particularly problematic when restaurants frequently update their menus, requiring new photos. Uber Eats aims to address these barriers through AI-driven solutions, enabling all merchants to present their offerings effectively.

A key challenge for Uber Eats is balancing the desire for enhanced photos with consumer demand for authenticity and a significant distrust of AI-generated content. Consumers prefer genuine, real-looking photos, and a meaningful fraction of them actively distrust anything created by AI. This distrust poses a risk of creating 'AI slop' that could degrade platform trust if not carefully managed.

To prevent marketplace homogenization and preserve individual merchant identities, Uber Eats designed strict constraints for its AI image editing. The AI must remain faithful to the original image and preserve the unique brand of each merchant. This approach avoids generic prompting that could lead to all restaurant listings looking the same, thus preventing a collapse in marketplace diversity.

The core goals for deploying AI agents in image optimization at Uber Eats are multifaceted. These include preserving authenticity and consumer trust, selectively improving image quality only where needed, and optimizing the global marketplace's visual consistency. Crucially, the deployment must be safe and enable continuous learning and improvement of the AI system over time.

Uber Eats navigates a spectrum of agent design, balancing deterministic systems with agentic creativity. Deterministic systems, while predictable, often prove brittle and lack scalability in dynamic environments. Conversely, agents offer creativity and adaptability but necessitate robust safety guardrails and a balanced design to prevent unintended outcomes. The challenge lies in harnessing agentic capabilities while maintaining control and reliability.

The image enhancement pipeline at Uber Eats begins with an 'image understanding and routing agent,' which leverages multimodality by asking a Large Language Model (LLM) to describe the photo's content. This understanding informs the routing logic, determining whether an image should be enhanced or skipped. If an image is routed for enhancement, a 'reflect' agent evaluates the edited output against predefined metrics, while a 'synthesize' agent continuously refines prompts based on feedback.

A QA agent then conducts a thorough review, comparing the original, edited, and an 'expert reference' image to identify any issues. If the QA agent detects problems, the system returns to the reflect agent for further prompt refinement or to the routing agent for re-evaluation, initiating a closed-loop feedback mechanism. This iterative process allows for self-correction and continuous learning, ensuring that the AI agents improve over time.

This closed-loop system, with its reflect and synthesize sub-agents, automates prompt optimization without human intervention, ensuring the system remains adaptive and self-improving. The process ensures that image enhancements are precise, relevant, and aligned with the platform's quality standards, minimizing the need for manual oversight.

Ultimately, the output is assessed for 'publish-readiness' through a series of QA gates, which include evaluating visual quality and checking for reward hacking. This comprehensive workflow ensures that only high-quality, authentic images are deployed, maintaining user trust and platform integrity.

Uber Eats has identified a form of 'reward hacking' where AI models make visually different but meaningless changes to images. This occurs when models prioritize minor pixel alterations over substantive improvements, a phenomenon described as 'nugatory changes.' Such conservative oversteering can happen as models attempt to avoid rejected outputs, leading to edits that are not genuinely meaningful or influential to the user experience.

Uber Eats sometimes encounters limitations from the underlying frontier models used for image editing, where issues like physics or object coherence can 'leak up' into their applied use cases. To address these upstream problems, Uber directly coordinates with the teams developing these frontier models. This collaborative approach ensures that foundational model issues are resolved, preventing them from impacting the quality of the image enhancements on the Uber Eats platform.

Multimodality is essential for achieving confidence in production, particularly for accurately identifying specific counts and details within images. If the system cannot reliably verify core details, such as the number of items in a dish, it will reject the output. This stringent verification process is critical to ensure accuracy and maintain trust in the AI-generated content on the platform.

Uber Eats employs a 'Swiss Cheese Model' for quality assurance, which involves multiple, redundant safety gates to minimize the chance of a failure reaching production. This approach optimizes for reducing the probability of errors. The final 'publish-ready' QA gate serves as a holistic safety net, encompassing various checks before an image goes live.

This multi-layered defense mechanism ensures that even if one gate has a 'hole' (a potential vulnerability), subsequent gates can catch the issue, preventing it from impacting the end-user experience. The model emphasizes redundancy and thoroughness, crucial for deploying AI agents at scale in a live marketplace.

By implementing this comprehensive QA framework, Uber Eats aims to guarantee the reliability and safety of its AI-enhanced content. This structured approach helps maintain the high standards required for customer satisfaction and merchant trust.

To enhance system-wide optimization, Uber Eats introduced the 'Diagnoser,' which acts as a higher-level abstraction for routing feedback to specific sub-agents within the overall AI system. The Diagnoser is designed to ingest various inputs from different feedback loops, then reflect on which particular agent needs optimization. This mechanism allows for more targeted and efficient tuning of configurations across multiple agents, improving the system's performance and adaptability.

Uber Eats continuously tunes its AI system by leveraging multi-source feedback, incorporating insights from both internal dogfooding and direct merchant feedback. This comprehensive approach ensures that the system evolves based on real-world usage and diverse user perspectives. The collected feedback is replayed against established benchmarks to validate improvements and confirm their effectiveness before full-scale deployment, maintaining a high standard of quality.

To achieve data-driven tuning at scale, Uber Eats meticulously tracks marketplace health through key metrics such as conversion rates. This data is then extensively sliced by various segments, including geography, device type, and dish type. This granular analysis allows the team to identify specific areas for improvement and tune the AI system for particular segments, optimizing performance across its diverse global operations.