Grok 4.6 Improves Benchmarks, Struggles in Design

xAI has notably shifted its development pace, transitioning from intermittent model releases to a rapid succession of updates.

This accelerated momentum is largely attributed to the recent acquisition of Cursor, which has brought new capabilities to xAI's development cycle. Grok 4.5, for example, received positive feedback for its practical utility in daily workflows.

The model excels at long-running tasks involving multiple sub-agents, consistently maintaining context over extended periods.

Users have reported that Grok 4.6 effectively manages task context, even with complex, multi-step instructions, ensuring tasks are completed without losing track.

Its Command Line Interface (CLI) is a high-quality integration point, enhancing its utility for developers and power users.

Despite its impressive capabilities, the model is not without flaws, and the reviewer noted minor bugs that indicate it is not yet a perfect solution.

Grok 4.6 represents a significant post-training update to its predecessor, Grok 4.5, rather than a new pre-training run.

This enhancement was made possible by xAI's integration of Cursor's reinforcement learning (RL) and advanced post-training infrastructure.

These improvements specifically target long-running agent reliability and enhance the model's performance in interactive and visual tasks, enabling it to manage complex multi-step workflows like research and coding.

Grok 4.6 achieved a substantial improvement, jumping from 54% to 66% on the DeepSeek benchmark, a significant increase for a post-training update.

On the Cursor Bench, Grok 4.6 surpassed Sonnet 3.5, overcoming previous concerns about data contamination, while its Frontier Code scores rose from 56.6 to 61.3, positioning it competitively between Sonnet and Claude 3.5 Sonnet.

During the initial launch week, subscribers are granted a 2x usage limit as the model integrates directly into the Cursor platform and the Grok build interface.

Improvements include a better optimizer and refined training recipes for the Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) stages.

xAI leveraged Grok 4.5 to regenerate SFT trajectories across key domains such as STEM, software engineering, and general knowledge work.

New capabilities allow the model to perform self-tests and verifications, enabling it to check its own work before proceeding with longer tasks.

xAI implemented additional safeguards to address potential hacking risks and the expanded agentic capabilities of Grok 4.6.

The company expanded its pre-deployment testing and incorporated third-party evaluation to precisely calibrate safety measures, including testing the model's ability to perform security audits on real-world codebases.

Grok 4.6 achieved a score of 61 on the Artificial Analysis Intelligence Index, representing a 5-point gain in just over a month and positioning it competitively with other frontier models.

The model demonstrated strong performance in specialized evaluations like the AA Briefcase and Harvey Lab benchmarks, prompting Elon Musk to suggest that Grok 4.7 will be significantly better and is expected within three to four weeks, incorporating massive amounts of SpaceX company data during its training.

This places Grok 4.6 competitively among other top-tier frontier models, according to artificial analysis.

Claude 3.5 Sonnet and Opus 3 outperform the model in UI design, where the reviewer found results plagued by excessive visual noise and poor card layouts.

In contrast, Claude 3.5 Sonnet and Opus 3 produced significantly better design work, leading the reviewer to conclude that Grok feels like a 'dumbed down' model in design contexts.

The reviewer noted that Grok 4.6's approach to design often results in outdated aesthetics and inefficient use of space.

The reviewer utilized a previous codebase, dubbed 'Fish Slop', to test Grok 4.6's capability in rebuilding complex software projects from scratch.

This method assesses the model's understanding of intricate, end-to-end software architecture and its ability to re-engineer existing solutions.

Kimi K3 outperformed the model on the task, as the reviewer found Grok 4.6 unable to scale game elements or properly manage UI highlights.

The game's pace and progression speed were found to be inadequate, with the reviewer noting that Kimi K3 performed significantly better on the same task.

Despite further attempts to fix the output, the model generated inverted controls and incorrectly placed objects, which the reviewer characterized as the worst 3D pass he had observed in a considerable period.

Open-weight models like Kimi significantly outperformed Grok 4.6 in this area, demonstrating a substantial 'generational gap' in their respective 3D capabilities.

It provided actionable advice for locking endpoints, suggested using the public suffix list, and recommended stopping the use of identity tokens in URLs, managing sandbox flags, all executed without errors during the audit.

Grok 4.6 was tasked with auditing a migration from ACP to the Cursor SDK, where it correctly identified technical limitations such as the absence of GUI-level approval and issues related to agent login.

However, the model failed to execute the migration effectively due to unexpected behavior, entering a 'plan mode' without providing a concrete plan, leading the reviewer to conclude that the current Grok implementation requires further refinement for agent-based coding workflows.

The model demonstrated its ability to plan and execute a substantial 1,000-line pull request, addressing multiple gaps in coverage for Grok Build.

The agent utilized custom skills to identify itself when replying on GitHub and successfully handled 'stacked' pull requests by analyzing local machine history and identifying gaps in event processing.

This highlights the inherent challenge for AI models to maintain context and manage competing multi-step instructions effectively in complex software engineering tasks.

Fable is recognized for its superior intelligence and orchestration capabilities, though it comes with a high cost and slower processing speed, while Soul is praised for its extreme thoroughness but is noted for generating excessive code.

Grok 4.5 was previously preferred for its superior speed and cost-efficiency compared to its peers, offering a balanced alternative in the market.

However, Grok 4.6, despite showing improved intelligence, exhibits regressions in both speed and cost-efficiency, indicating a trade-off in its latest iteration.

The host observed that while Grok 4.6 is not his preferred model, the rapid rate of intelligence growth demonstrated by xAI keeps him optimistic about future releases.

Elon Musk's focus on Grok 4.7 suggests significant upcoming improvements, underscoring the industry's need for increased competition to drive down costs and enhance overall model quality.