Perceptron AI Unveils Embodied Foundation Models

Armen Aghajanyan, CEO of Perceptron, explained the company's research stance, which is to move away from existing distinctions such as VLMs and world models.

The core mission of Perceptron is to create physical AI systems that can perceive and interact with the physical world in real-time, aiming to provide intelligence to various sensors, devices, and robots.

Aghajanyan highlighted his six-year tenure at FAIR, where his focus was on scaling up recipes for multimodal models.

He noted that traditional approaches to multimodal modeling typically rely on early fusion to combine diverse modalities, evolving from text-outputting VLMs to action-oriented VLA models and generative world models that represent entire environments holistically.

A unified architecture merges perception, embodied reasoning, and control into a single framework that handles cross-modal inputs and multi-domain outputs, according to Aghajanyan.

This model handles cross-modal reasoning as input and produces multi-domain outputs simultaneously, offering a comprehensive solution for complex AI tasks.

Computational inefficiency arises because only 2% of the 1 million visual tokens generated from an hour of video receive loss calculation, often ignoring critical elements like contact points.

Synthetic labeling and transcripts, while used, provide sparse and frequently incorrect signals, leading to inefficiencies.

Predicting every pixel indiscriminately wastes computational power by failing to prioritize critical elements such as gripper tips or contact points, which are more relevant to a robot's interaction with its environment.

Current robotics often relies on hard-coded percepts, such as the specific coordinates of gripper tips, which limits adaptability.

Perceptron AI aims to enable models to automatically learn semantic, unique objectives without requiring manual intervention, and Aghajanyan confirmed his team has developed a method for this, though specific details remain proprietary.

Robots equipped with always-on cameras face substantial computational challenges due to the massive volume of tokens generated from continuous video streams, a problem referred to as context bloat.

Unlike dense text data, video is inherently sparse, necessitating architectural breakthroughs to handle this imbalance natively rather than through synthetic means, as robots must reason over very long sequences of tokens without interruption.

Armen Aghajanyan states that text, image, and video tokens should not be treated identically, advocating for more nuanced processing.

He notes that while spatial compression methods, such as patch-wise averaging, can achieve up to 10x compression, they are considered a temporary solution.

Perceptron AI released a paper on their Data Sparse Mixture of Experts approach, which employs a router within the model to predict which tokens to input and which to skip, allowing the model to automatically focus on high-density, task-relevant information.

Trained on a 1-petabyte dataset of internet crawls and synthetic data, the new model released by Perceptron AI functions at roughly 15 times the cost efficiency of competing systems while matching Gemini 1.5 Pro performance.

The new model is significantly more cost-effective, being approximately 15 times cheaper than competing embodied reasoning systems, and was trained on a 1-petabyte dataset that includes internet crawls, custom recipes, and synthetic data, covering text, images, videos, and various trajectories, including desktop use and gaming.

Perceptron's model operates as an agent capable of generating code, zooming into areas, and adjusting contrast to solve complex visual problems where traditional box detection methods fall short.

This agentic approach allows the model to autonomously decide when and how to process image segments, enabling it to identify challenging objects, such as birds in varying visual contexts.

Armen Aghajanyan introduced a distinction between 'embodied reasoning models,' also known as orchestrators, and 'tactile policy models' to manage complex robotic tasks.

Orchestrators are designed to break down high-level tasks, such as making coffee, into smaller, manageable subtasks for execution, while the full spectrum of agentic systems ranges from pure VLA models to sophisticated high-level reasoning modules, with embodied reasoning remaining a significant unsolved problem in robotics.

Perceptron's models autonomously navigate through video footage, clip relevant segments, and verify captions, demonstrating self-verification capabilities.

These frontier embodied reasoning systems achieve significant efficiency gains, performing complex processing at a fraction of the cost associated with models like Gemini, and possess capabilities previously unseen in other AI architectures.

Perceptron has discovered new scaling laws that enable a strategic balance between control-based, trajectory, perceptive, and reasoning training data.

Teleoperation data is expensive, costing approximately $100 per hour, in contrast to video pre-training data, which is significantly cheaper.

The model demonstrates a 10x trade-off ratio, meaning that 10 times more video pre-training data can effectively replace 10 times less teleoperation data, leading to substantial performance benefits over training pure VLAs.

Identifying objects and allocating them into bins requires a multi-step integration of perception and control, a complexity that forces the industry toward joint modeling.

An example task involves identifying objects, such as books, and then properly allocating them into designated bins, which requires a complex interplay of perception and control, suggesting that joint perception and control modeling is essential for advanced robotic actions.

Refining how models track key frames against delta frames will determine whether robots can effectively manage spatial relationships like 'left' and 'right' during live deployment, the speaker explained.

Effective context management involves techniques such as distinguishing between key frames and delta frames, allowing models to efficiently process and manage information.

Training models to recognize spatial relationships, like 'left' and 'right,' is critical for robotics, and by targeting specific data distributions early, Perceptron's models outperformed Gemini ER with less computational power.

Traditional VLA models frequently fail when environmental factors, such as background or lighting, undergo slight changes, highlighting a significant limitation.

Joint perceptive and control modeling substantially increases robustness against variations in backgrounds, demonstrating greater resilience in diverse settings, though challenges remain with extreme lighting conditions.

Traditional models often fail when minor changes occur in background environments, such as altering a table surface, leading to breakdowns in robotic tasks.

Jointly modeling perception and control significantly enhances stability against lighting variations, providing greater resilience, but extreme lighting conditions, such as flashlight glare, continue to pose challenges, underscoring the ongoing need for advancements in robustness to background noise.

Perceptron AI employs online augmentations during training to simulate camera failures and artificial variations like sunlight from specific directions, aiming to improve model robustness.

The significant performance gains in robotics are attributed to adapting early fusion paradigms from other domains and integrating them into the robotics domain.

Perceptron's K1 model excels at deep structured extraction for complex egocentric annotation, rather than building formal ontologies, proving highly effective at this task.

The platform's models are proficient in captioning images and videos, and all demonstrated tools, APIs, and benchmarks are publicly available for developers to use.