Grok 4.7 Falls Short of Elon Musk's Bold Claims

Musk specifically highlighted SpaceX's unique training corpus as a decisive advantage, suggesting it would make Grok 4.7 unrivaled in real-world coding performance. However, preliminary testing indicates that Grok 4.7 is currently underperforming compared to its predecessor, Grok 4.6, across several benchmarks.

Traditional benchmark metrics and scoring systems are no longer accurately representing the practical utility and performance of AI models in real-world applications.

The sheer volume of misleading benchmark data surrounding recent model releases, including Grok 4.7, has caused considerable frustration among developers and evaluators.

Despite its benchmark scores, Grok 4.7 remains a personal favorite for the content creator due to its specific real-world coding performance, which often diverges from synthetic test results.

However, data from Artificial Analysis indicates that Grok 4.7 uses 81,000 tokens per task, a substantial increase compared to Grok 4.6's 38,000 tokens.

This token usage is also considerably higher than that of other competitive models like Fable 5.1, making the promise of improved token efficiency effectively inaccurate.

Analysis using Artificial Analysis benchmarks shows a notable increase in the operational cost of Grok 4.7.

Grok 4.6 cost $1.86 per task, Astra cost $3.26, and Grok 4.7 was the most expensive at $3.74 per task, representing roughly a twofold increase in cost for the newer model in these specific runs.

Michael from XAI explained that in production environments at SpaceX and Cursor, Grok 4.7 would use 5% more tokens for median requests and 20-30% more for P99 requests compared to 4.6.

XAI attributes these differences to a distribution shift in how users formulate prompts for the model.

The speaker questioned XAI's assertions regarding token efficiency and the underlying model architecture, suggesting they might be misleading.

Doubts were raised about whether the model truly utilizes a new 2.1 trillion parameter base model as claimed.

It was also argued that measuring token usage by 'request' is an imprecise method, as prompts can involve a variable number of tool calls, making a single request an inaccurate unit of measure.

A private benchmark conducted by the speaker revealed that Grok 4.7 incurred a cost of $20 in its most expensive run, significantly higher than other models.

Most other models did not exceed $10, with the sole exception of one Astra run that barely reached $11.44, while Opus 5 runs were observed to be less than a quarter of Grok 4.7's price.

The speaker noted the absence of Astra in the official XAI benchmarks and questioned its exclusion.

OpenAI's policies prohibit SpaceX from including OpenAI models in their competitive benchmarks, which explains Astra's omission.

Grok 4.7 was specifically trained to natively understand the Grokbot harness, which the speaker initially found unusual but later recognized as a genuinely beneficial feature.

Despite initial skepticism, this integration has proven to be highly advantageous and popular among users, making Grokbot a legitimate tool.

Models exhibit random performance spikes and drops across different reasoning levels, rather than the smooth, predictable curves expected from other benchmarks, with Grok 4.7 trailing Grok 4.6 due to 'overthinking' on certain tasks.

Remarkably, Grok 4.7 scored lower than Muse Spark 1.3 across all Artificial Analysis benches, indicating an underwhelming overall performance.

Even a six-point difference in benchmarks can significantly indicate how frequently a model might make errors or produce suboptimal outputs within a given timeframe.

Reducing the error rate effectively extends the usable time window for completing tasks, as users spend less time correcting mistakes.

Higher-tier models minimize the need for extensive 'hand-holding' or detailed upfront specifications from the user, offering a more autonomous experience.

In contrast, Grok 4.7 demands more user effort to achieve the same level of quality and results as its top-tier competitors.

Artificial Analysis observed Grok 4.7 achieving third place in AA Briefcase benchmarks, demonstrating significant gains in specific areas.

Despite these improvements, Grok 4.7 showed a regression in presentation quality, with its ELO score actually lower than Grok 4.6, while Anthropic remains the leader in design capabilities, and Astra has shown improved steerability.

The release of Grok 4.7 was delayed from the previous week due to issues related to Reinforcement Learning (RL) tuning.

Elon Musk attributed these delays to excessively penalizing response length during training, which caused the model to abandon difficult tasks prematurely, implying that fixing these behaviors likely compromised the previously touted token efficiency.

The speaker evaluated various models on their ability to identify and suggest improvements within the T3 Code codebase.

Astra demonstrated superior performance, identifying the most novel improvements and thoroughly vetting its suggestions, with Grok 4.7 performing surprisingly well, just behind Astra in the viability of its suggestions.

Fable 5.1, while making fewer suggestions, maintained high quality and effectively avoided generating unhelpful or incorrect paths.

The evaluator also noted some confusion regarding the versioning of Gemini 37 versus 38 during the process.

The economic benefits once associated with Grok 4.5 are no longer present, as Grok 4.7 is slower, more expensive, and exhibits higher error rates compared to its competitors, suggesting it might only serve as a transitional 'stepping stone' for future iterations.

Fable and Astra represent a significant generational leap in AI technology, setting new standards for performance.

Grok 4.7 is characterized as 'last gen' in performance while being priced at 'current gen' rates, making it increasingly difficult to justify as the AI frontier continues to advance, with the host hoping for future RL improvements to enhance model efficiency, reserving recommendation only for those committed to the Cursor ecosystem.