Frontier AI Models Fail Visual Reasoning

A significant performance gap exists between the visual reasoning of humans and models such as Claude, ChatGPT, and Gemini, according to Elorium CEO Andrew Dai.

Dai contends that these widely used models are a considerable distance from meeting any practical definition of Artificial General Intelligence (AGI) when it comes to visual tasks.

Tasks like counting chessboard pieces often result in hallucinated objects because current models rely heavily on pattern matching instead of spatial analysis.

For instance, when shown a chessboard, models incorrectly assume a full board structure and count 32 squares, relying on common patterns instead of actual visual evidence.

This pattern-matching mechanism, while effective for simple object classification like plants and animals, proves detrimental when complex spatial analysis is required.

When presented with a board game and asked to count rows, AI models fail to perform accurate counting and instead provide estimations that defy logical observation.

Andrew Dai emphasizes that current frontier models lack spatial grounding, rendering them unable to answer questions that demand detailed visual analysis and understanding.

In video-based tasks, AI models often fail to track critical state changes, such as a robot arm lifting a lid or activating a stove.

This phenomenon, termed 'context amnesia,' arises because models struggle to maintain consistency and track events across extended video sequences, easily losing sight of ongoing actions.

Andrew Dai categorizes rapid recognition as System One thinking and deliberate, step-by-step analysis as System Two, noting that current models struggle with the latter.

He proposes that any visual task taking humans longer than a second to solve is a complex reasoning task, not mere recognition.

While frontier models excel at quick recognition, they consistently fail when faced with tasks demanding detailed, step-by-step analytical processes.

Existing benchmarks for AI visual tasks frequently rely on low-resolution images, typically 32x32 or 64x64 pixels.

Andrew Dai challenges the notion that complex, real-world tasks can be effectively reduced to such limited resolutions, arguing that most meaningful visual problems require far greater detail. While MMMU improves upon MMLU by incorporating images, many questions within it can still be answered without genuine visual comprehension.

Geometric alignment and object permanence remain unaddressed in visual AI, as industry adoption skews toward text-based applications and coding.

Andrew Dai asserts that a critical missing paradigm in visual AI is 'thinking,' leading to a dearth of benchmarks for crucial areas such as geometric alignment, spatial intelligence, and object permanence.

While tools like Google Lens, SAM 3, and YOLO offer robust recognition capabilities, they remain passive and lack active reasoning or interaction with their environment.

True advancement in visual AI necessitates the development of 'thinking visual models' capable of active spatial and temporal intelligence, moving beyond mere generation and recognition.

Andrew Dai discusses Elorium's four-stage approach to developing visual reasoning, which includes custom multimodal data collection, a synthetic data flywheel, and architectural enhancements to Transformers.

He points out that existing frontier models primarily use textual chain of thought, thereby overlooking crucial visual reasoning processes.

A natively visual chain of thought enables models to process steps visually, such as drawing bounding boxes around objects before accurately answering a counting question.

Elorium has expanded its team to approximately 20 members, strengthening its capabilities in AI development.

Dustin Tran, formerly the lead of post-training at XAI, has joined Elorium as the chief reasoning architect. The team also includes professionals from leading tech companies such as Apple, DeepMind, and Amazon.

Andrew Dai states that existing vision models are trained on static images and directed videos, consequently lacking the active physical interaction needed for dynamic robotic tasks.

He notes that robots currently face significant challenges with real-time physical actions in complex, dynamic environments, often relying on over-engineered and brittle existing methods. Elorium plans to release a model API by the end of the year to integrate action-relevant scene understanding into robot control systems.

Construction sites enforce zone-specific safety regulations, which conventional computer vision struggles to adapt to quickly.

Andrew Dai explains that current methods necessitate frequent retraining due to the inherent brittleness of existing models, making it difficult to interpret policy language like OSHA's and match it to real-time site conditions. Elorium aims to ground video streams in safety regulations to interpret policy compliance using existing camera infrastructure.

Designing even a small part of a robot testing platform can consume 100 to 200 hours, with an entire platform requiring 2,000 to 3,000 human hours.

Frontier models struggle with visual context in CAD, CAM, and blueprints, making the design process inefficient. Elorium addresses this by using programmatic and simulation validation to verify designs, mirroring how code is rigorously tested. The future of mechanical design, encompassing advancements in cars, rockets, and batteries, increasingly depends on AI-driven visual reasoning.

In architecture, precise counting and spatial awareness are paramount; an error in the number of bedrooms, for instance, leads to significant real-world repercussions.

Mechanical engineering firms report that designing small parts for robot platforms takes 100-200 hours, while developing an entire testing platform can consume 2,000 to 3,000 human hours. Current frontier models consistently fail to accurately interpret visual data from 3D CAD, CAM files, and blueprints, leading to frequent and costly errors in the design process.

Their approach uses multimodal reasoning to extract geometric logic, enabling programmatic or simulation-based validation. While mechanical design shares similarities with coding—both benefiting from automated testing—hardware design, such as rockets or phones, remains fundamentally visual and currently less reliant on AI acceleration.

Quality control for CAD and CAM files represents a primary potential use case for advanced AI, ensuring precision in engineering designs.

Andrew Dai envisions AI as a crucial future component that will empower engineers to develop faster cars, more efficient rockets, and improved batteries. He argues that the inherent visual and spatial complexity of physical product development means these advancements cannot be achieved through software coding alone.