Designing Multimodal Collaborative Commerce Agents

Nidhi Kaushik Vyas introduced multimodal collaborative agents as systems designed to understand and act upon vague user intentions, particularly in shopping scenarios where users lack specific keywords. These agents are characterized by their high agency execution and proactive elicitation of user preferences.

These frameworks are not limited to commerce but can also be applied to fields like finance and education, focusing on situations where users' initial queries are imprecise.

Current search agents often function as mere wrappers around a search bar, assuming users have a well-defined intent. However, users frequently begin their shopping journeys with an unclear feeling or 'vibe,' leading to a significant articulation gap.

An effective agent must be able to bridge this gap, helping users translate their vague ideas into concrete preferences that can be acted upon.

The proposed agent framework operates through a three-stage shopping loop: Discovery, Research, and Response. The Discovery phase focuses on gathering context and forming an initial strategy based on the user's input.

The Research phase involves eliciting more detailed preferences from the user and performing the necessary 'heavy lifting' to understand their needs. Finally, the Response phase is where the agent delivers information in an adaptively presented format to optimize the user experience.

To overcome the limitations of text-based preference elicitation, agents need to become more creative in how they engage with users. This includes using visual references and inspiration boards to help users articulate subjective preferences.

By employing visual tools, the agent can better ground its understanding of the user's desires, handling complex tradeoffs and summarizing information effectively.

Many existing systems often default to text-heavy responses, which can be overwhelming for users. Instead, an intelligent agent should adapt its response format to the user's specific query and underlying intent.

This adaptation means using bulleted lists for clear information, comparison tables for detailed analysis, or visual boards for stylistic inspiration, thus demonstrating smarter presentation skills.

In the Discovery phase, the agent begins by aggregating context from various signals. These signals include past conversations, reference images provided by the user, and external links.

The agent also integrates personal context relevant to the user to construct a comprehensive 'working state' that informs subsequent actions.

The agent's internal working state includes maintaining a detailed session history to keep track of the interaction flow. It also involves extracting relevant user context from all available inputs.

Furthermore, the agent identifies and differentiates between hard constraints (non-negotiable requirements) and soft constraints (flexible preferences) that the user might have provided in their query.

Managing variables in real-time is crucial for the agent to pull out and display relevant information, as these variables directly affect the results presented to the user. This dynamic management ensures that the information remains current and accurate.

It helps avoid stale information by frequently refreshing inventory and other dynamic data, thereby impacting the quality and relevance of the displayed results.

A key aspect of agent intelligence is discovering the 'intent gap,' which refers to the unknown variables that prevent the agent from providing the best possible answer. Often, users lack the precise terminology or knowledge to fully articulate their needs.

The agent's task is to identify this missing information and manage these unknown variables efficiently to maximize information gain with minimal user effort.

The agent needs to develop a collaborative strategy, evaluating all possible next moves to determine which unknown variable to prioritize. This involves comparing different questions or actions the agent could take to elicit more information.

The goal is to prioritize queries that offer the maximum information gain without over-asking the user. This is achieved by measuring the utility of each potential question.

The agent dynamically assesses which piece of information is most critical at any given moment, ensuring a smooth and efficient interaction flow that progressively refines its understanding of the user's fuzzy intent.

A crucial initial step for the agent is to establish a temporary bridge between the user's constraints and the product catalog, which forms part of the knowledge base. This involves real-time mapping of explored constraints to the underlying ontology.

This process ensures connectivity and consistency between the user's expressed intent and the available products and services within the database.

The agent intelligently determines the best method for eliciting information, particularly when dealing with subjective user constraints. For such preferences, purely textual elicitation is often insufficient and less effective.

Instead, the agent may opt for visual preference boards or other interactive formats to better engage the user and gather nuanced insights that text alone cannot capture.

The agent continuously observes user reactions through micro-signals to refine its confidence model. These micro-signals include actions like hovering over certain elements or clicking in specific directions.

By tracking these behavioral cues, the agent updates its internal confidence model, allowing it to adapt its understanding and responses based on real-time user interaction.

The final step involves using model intelligence to determine the optimal response format based on user intent and contextual metadata. This ensures that the information is presented in the most consumable and effective manner.

For instance, policy or review information might be best conveyed through bulleted lists, while trade-off analysis benefits from comparison tables. For creative style explorations, visual inspirations or mood boards are more appropriate.

This adaptive approach ensures that the user receives information in a format that directly addresses their needs and facilitates their decision-making process, moving beyond generic text responses.

Effective AI commerce agents are evaluated based on several key metrics. First, information visibility is crucial: data must be easy for users to spot and comprehend, enabling them to make quick decisions and progress through their purchasing journey.

Second, data fidelity is paramount to prevent 'hallucinations' or inaccuracies, ensuring the agent provides reliable information. Finally, user actionability means the response structure should inherently prompt the next logical step in the user's interaction, guiding them smoothly towards their goal.

The system heavily leverages the domain expertise of merchants to structure retrieval and ensure fairness. This involves collaborating with merchants to map user constraints to specific metadata within their product ontology.

The Universal Commerce Protocol (UCP) allows merchants to share a common language with the agent, facilitating the accurate and fair retrieval of products based on user preferences and merchant data.

The agent's intelligence, particularly regarding response format, is centralized and controlled by the agent itself, not the merchant. While merchant input informs the agent, the agent makes the final decisions on presentation to ensure a consistent and horizontal user experience across all merchants.

Users often express a strong preference for active involvement during the discovery and inspiration phases of their shopping journey, often referred to as 'upper funnel' interactions. In these stages, users prefer to interact directly with the system rather than relying solely on automated agent-to-agent interactions.

This suggests that human-like interaction remains highly valued for exploration and nuanced decision-making, as opposed to purely transactional tasks where automation might be more welcome.