Uber Eats Details Multimodal AI Agent Deployment Strategy for Image Optimization

The platform is active in 10,000 cities globally, connecting millions of restaurants with customers. This extensive reach underscores the impact and complexity of deploying AI solutions across its diverse marketplace.

Image quality is a critical factor for delivery platforms like Uber Eats because photos serve as the initial point of contact for customer impressions. High-quality images directly influence click-through rates and cart conversion rates, making them vital for driving sales. Furthermore, the importance of video content is also on the rise, indicating a broader trend toward richer visual engagement.

This issue becomes particularly problematic when restaurants frequently update their menus, requiring new photos. Uber Eats aims to address these barriers through AI-driven solutions, enabling all merchants to present their offerings effectively.

This distrust poses a risk of creating 'AI slop' that could degrade platform trust if not carefully managed.

The AI must remain faithful to the original image and preserve the unique brand of each merchant. This approach avoids generic prompting that could lead to all restaurant listings looking the same, thus preventing a collapse in marketplace diversity.

These include preserving authenticity and consumer trust, selectively improving image quality only where needed, and optimizing the global marketplace's visual consistency. Crucially, the deployment must be safe and enable continuous learning and improvement of the AI system over time.

Deterministic systems, while predictable, often prove brittle and lack scalability in dynamic environments. Conversely, agents offer creativity and adaptability but necessitate robust safety guardrails and a balanced design to prevent unintended outcomes. The challenge lies in harnessing agentic capabilities while maintaining control and reliability.

This understanding informs the routing logic, determining whether an image should be enhanced or skipped. If an image is routed for enhancement, a 'reflect' agent evaluates the edited output against predefined metrics, while a 'synthesize' agent continuously refines prompts based on feedback.

A QA agent then conducts a thorough review, comparing the original, edited, and an 'expert reference' image to identify any issues. If the QA agent detects problems, the system returns to the reflect agent for further prompt refinement or to the routing agent for re-evaluation, initiating a closed-loop feedback mechanism. This iterative process allows for self-correction and continuous learning, ensuring that the AI agents improve over time.

This closed-loop system, with its reflect and synthesize sub-agents, automates prompt optimization without human intervention, ensuring the system remains adaptive and self-improving. The process ensures that image enhancements are precise, relevant, and aligned with the platform's quality standards, minimizing the need for manual oversight.

Ultimately, the output is assessed for 'publish-readiness' through a series of QA gates, which include evaluating visual quality and checking for reward hacking. This comprehensive workflow ensures that only high-quality, authentic images are deployed, maintaining user trust and platform integrity.

Uber Eats has identified a form of 'reward hacking' where AI models make visually different but meaningless changes to images. This occurs when models prioritize minor pixel alterations over substantive improvements, a phenomenon described as 'nugatory changes.' Such conservative oversteering can happen as models attempt to avoid rejected outputs, leading to edits that are not genuinely meaningful or influential to the user experience.

Uber Eats sometimes encounters limitations from the underlying frontier models used for image editing, where issues like physics or object coherence can 'leak up' into their applied use cases. To address these upstream problems, Uber directly coordinates with the teams developing these frontier models. This collaborative approach ensures that foundational model issues are resolved, preventing them from impacting the quality of the image enhancements on the Uber Eats platform.

If the system cannot reliably verify core details, such as the number of items in a dish, it will reject the output. This stringent verification process is critical to ensure accuracy and maintain trust in the AI-generated content on the platform.

This approach optimizes for reducing the probability of errors. The final 'publish-ready' QA gate serves as a holistic safety net, encompassing various checks before an image goes live.

This multi-layered defense mechanism ensures that even if one gate has a 'hole' (a potential vulnerability), subsequent gates can catch the issue, preventing it from impacting the end-user experience. The model emphasizes redundancy and thoroughness, crucial for deploying AI agents at scale in a live marketplace.

By implementing this comprehensive QA framework, Uber Eats aims to guarantee the reliability and safety of its AI-enhanced content. This structured approach helps maintain the high standards required for customer satisfaction and merchant trust.

The Diagnoser is designed to ingest various inputs from different feedback loops, then reflect on which particular agent needs optimization. This mechanism allows for more targeted and efficient tuning of configurations across multiple agents, improving the system's performance and adaptability.

This comprehensive approach ensures that the system evolves based on real-world usage and diverse user perspectives. The collected feedback is replayed against established benchmarks to validate improvements and confirm their effectiveness before full-scale deployment, maintaining a high standard of quality.

To achieve data-driven tuning at scale, Uber Eats meticulously tracks marketplace health through key metrics such as conversion rates. This data is then extensively sliced by various segments, including geography, device type, and dish type. This granular analysis allows the team to identify specific areas for improvement and tune the AI system for particular segments, optimizing performance across its diverse global operations.