Meta Engineers Detail New Video Processing AI

The approach involves building specialized small-scale agents that contribute to a larger multi-agent system, moving beyond simple precision and recall to incorporate LLM-level pipelines and workflow optimization due to the dynamic, messy, and drift-prone nature of real-world video data.

Intra-modality issues currently rely on cosine similarity, but Aditya Gautam distinguishes them from inter-modality misalignments, which require granular, frame-level analysis to detect unexpected shifts to unrelated material.

Inter-modality misalignment, however, refers to unexpected shifts in video content, like the sudden appearance of unrelated advertisements or political material, requiring granular, frame-level analysis of vision and video content to detect such adversarial behaviors.

The proliferation of AI tools has significantly lowered the barrier for duplicating and transforming content at scale, making it easier for users to copy and modify videos.

This ease of duplication leads to an imbalance in content attribution and user fatigue from seeing repetitive videos, making it critical to trace the original source of transformed content to maintain the health of the content ecosystem.

A multi-agent approach is preferred over a single large language model (LLM) due to the inherent complexity involved in content retrieval, reasoning, and comprehensive video analysis.

The 'Reviewer Agent' functions as the centralized orchestrator and API gateway, managing the overall workflow and interactions between different components.

The 'Perceiver Agent' is a sophisticated Vision-Language Model (VLM) expert, responsible for breaking down video content through temporal change detection rather than fixed frame rates, thereby compressing data efficiently for further analysis.

By aggregating temporal metadata into a JSON object, the Perceiver agent facilitates temporal analysis, allowing the system to identify modality misalignments or transitions between distinct topics like sports and politics.

This agent aggregates temporal metadata into a JSON object, enabling temporal analysis to detect content shifts, like transitions between different topics, and helps identify anomalies or potential modality misalignments within the video content.

The Retriever agent processes signals and metadata from the Perceiver agent, adaptively indexing this information into various databases to optimize for different types of queries.

Topics are stored in inverted indexes, embeddings are managed using vector databases, and extracted entities are organized within graph databases, allowing for efficient online inference to query similar clips and entities.

Operating at a massive scale, processing billions of frames, necessitates moving beyond standard VLM models by implementing techniques like model compression, knowledge distillation, and quantization to ensure cost-effectiveness.

Pre-training and fine-tuning are crucial steps for deploying specialized VLM models tailored to specific problems, with offline analysis often leveraging frameworks like Ray clusters for big-scale data processing.

Beyond static VLM analysis, the Reviewer agent integrates real-time user feedback—such as likes, dislikes, reports, and comment sentiment—to dynamically interpret how an audience responds to content.

Sentiment analysis from user comments provides dynamic feedback, allowing the system to understand the video's reception and adapt its response based on real-time changes in sentiment rather than relying solely on semantic content.

Training on domain-specific image and language tokens enables superior fine-tuning for in-house workflows, contrasting with foundational models typically built on clean, curated web data.

Pre-training these models on domain-specific image and language tokens, despite the high cost, yields substantial performance improvements during fine-tuning, allowing them to better adapt to the unique characteristics of internal datasets.

Instruction fine-tuning leverages specific internal policies and guidelines to train models, enabling them to produce structured outputs, such as JSON schemas, for easier validation of modality misalignment or content duplication scores.

The system architecture integrates a vision encoder, a projector acting as a bridge, and an LLM, providing context including roles, policies, and metadata to guide the agent's behavior effectively.

Direct Preference Optimization (DPO) techniques are applied to align models with specific domain policies, while a sub-sample of production data is processed through an LLM judge.

Human reviewers are then engaged to examine cases where models fail to meet prediction thresholds, helping to identify if errors stem from chain-of-thought reasoning or retrieval failures.

This continuous improvement loop involves creating new datasets from negative samples to retrain models and actively combat model drift, ensuring sustained performance.

By favoring domain-specific tasks over general-purpose frontier models, engineers achieve scalability through knowledge distillation and quantization while balancing inference costs against accuracy.

Internal domain-specific tasks do not require the broad, general-purpose capabilities found in frontier models, making knowledge distillation and quantization essential for optimizing inference costs.

Engineers must develop performance tables that balance model size, compute savings, and accuracy to ensure systems are not only production-ready but also scalable and cost-effective beyond merely solving the initial problem.

Retrieval latency, reasoning quality at the agentic level, and Chain of Thought efficiency provide a more granular view of system performance than traditional binary metrics.

Assessment includes crucial factors such as retrieval latency, the quality of reasoning at the agentic level, and the efficiency of the Chain of Thought process, which helps identify specific failure points like retrieval errors, tool call issues, or LLM hallucinations.

The implementation of adaptive reasoning budgets allows for a dynamic trade-off of compute resources based on the complexity of the problem, while monitoring total token cost and latency per agent ensures overall system efficiency.

LLM-as-a-judge models, used for evaluation, must be continuously monitored for drift against human evaluation benchmarks, with retraining on new datasets from labeling teams as needed.

System optimizations include spatial-temporal reduction for compressing redundant frames, caching viral content to bypass multi-agent systems when high similarity scores are detected, and metadata pruning to filter content from authentic creators before it reaches expensive compute layers.

Decomposing problems only when necessary and prioritizing adaptive optimization for ROI are fundamental, as effective evaluation serves as the bedrock for all production monitoring.