AI's Rapid Evolution: Performance, Engineering, and Market Dynamics

Users report that Claude's Soul model, despite being available in the plus plan, has become noticeably less capable, with some calling it the 'dumbest model you've seen on Earth.'

The host expresses uncertainty regarding the actual value proposition of the $20 subscription plan for users. He notes that he does not have these subscriptions and therefore lacks direct experience with their specific usage patterns.

Theo argues that current regulatory proposals are structured to primarily affect 'Frontier Labs' rather than smaller, emerging companies. He states that he does not see validity in the 'regulatory capture' argument as it is currently presented, noting that he would be open to the conversation if specific, evidence-based strategies were cited. He adds that David Sachs is someone he considers historically prone to making claims without concrete proof.

He mentions that he will be a little rude about it, expressing frustration that people suggest overly simplistic fixes for complex engineering tasks.

He clarifies that their team is already running a second database to manage the transition process. He emphasizes that if you download it now, this is how it works, as they have clearly considered the alternatives to what are complex engineering challenges.

Performance updates have addressed the historically slow, memory-intensive nature of Claude's interface, according to the host who has tracked these improvements. He notes that this has been one of his favorite things to dunk on them for for years at this point.

He notes that the models are becoming effective enough to help Anthropic improve its own software. He references Anthropic's claims that their interface, including the desktop app, has become as much as three times faster.

This progress is particularly interesting given his prior experiences, including performance issues in his own T3 code desktop app, some of which were influenced by Fable 5 models. He mentions that he had to go and fix them himself because he found that Fable 5 was not the best at digging deep and fixing performance issues, though 5.1 and Opus 55 seem better.

During a live test, Theo observes that deleting threads does not immediately remove them from the sidebar, indicating that the interface is not effectively invalidating the cache. He notes that the UI continues to list threads that no longer exist, causing them to appear broken until they eventually vanish after further interaction.

Anthropic utilized 'Claude Tag,' a Slackbot, to conduct its performance improvements, rather than Claude Code. The host mentions that he has seen examples from the person who started Claude Tag doing all of his work through it.

They used an internal research model described as roughly comparable to Opus 5.5. Theo notes that internal testing often benefits from having fewer guardrails, which he knows is a good call out because when diving into Chrome internals, it is much more likely that you start to hit safety guards.

He adds that he is hopeful because he has not pushed Opus too hard yet, though he did recall an instance where another agent hit some type of security flag or guardrail when it was working on performance improvements. He emphasizes that he stays in T3 Code 99% of the time, but he was in the terminal because he was making changes to T3 Code.

Theo recounts an experience where a 'boil the ocean' prompt, designed to encourage extreme solutions for LakeBed's performance, led to a custom runtime and a 2 to 4x performance improvement.

This approach pushed the AI to propose unconventional solutions, including forking Rusty V8 to build a custom runtime, which significantly enhanced performance.

He emphasizes that giving AI agents explicit permission to explore 'wacky' and non-standard ideas is crucial for achieving radical architectural innovations.

The planning phase for this complex task took 20 minutes, followed by a 2-hour initial build, and then 5.5 hours for the AI to complete the PR, demonstrating the extensive effort involved in such projects.

Theo warns that relying solely on automated PRs without manual verification can lead to undetected software regressions that users might not report.

He advocates for building tools that enable developers to easily trigger instant builds, such as DMG files for desktop applications, to facilitate personal testing of changes.

He argues that minimizing friction in the testing process is as vital as measuring performance metrics, as excessive friction discourages developers from performing thorough manual checks.

He highlights that many companies, from small startups to large businesses, struggle with this, often enduring 15 to 25 minute processes just to test a single change in a PR.

This discovery highlighted an inefficiency where nearly 7,000 hooks were triggering recalculations for the composer functionality.

Additionally, Claude identified a single root 'has' selector that added 24 milliseconds to every DOM change.

These findings underscore the necessity of deep technical auditing in complex codebases.

Chrome's background pre-rendering process, which triggers as users type a URL, resulted in a mismatch between the initial buffer size and the rendered page.

Chrome draws its own 56-pixel footer on new tab pages, and when a tab navigates to a new site, this footer disappears, making the page 56 pixels taller 100 milliseconds after the first paint.

The mismatch between the pre-rendered page's size and the final view on navigation caused these shifts, rather than issues with the static composer or Chrome extensions.

The speaker highlighted the AI's ability to deduce such browser-level behaviors as a significant breakthrough, enabling a deeper understanding of complex interactions.

Theo emphasizes that models improve significantly when instructed to first measure and validate performance before attempting to improve it.

This strategy avoids the pitfall of hoping the model will correctly identify and fix issues on its own, instead ensuring a structured approach to improvement.

Working with and around 'slop,' or AI-generated inefficiencies and errors, is becoming a skill in modern engineering practices.

This shift in methodology necessitates the development of new systems and structures to guide AI agents toward desired outcomes.

Cloudflare demanded a minimum annual commitment of $50,000 to access specific tunnel pricing for T3 Connect, despite the service advertising free tunnels to signed-out users.

Theo criticized the irony of Cloudflare promoting free access on one hand while pressuring startups like his for substantial enterprise contracts on the other.

He expressed reluctance to sign contracts that would consume a significant portion of his startup's payroll, stating that such a commitment was not feasible.

This ongoing negotiation highlights the friction many smaller companies face when dealing with enterprise sales models that lack public pricing transparency.

Theo responded to criticisms regarding his past technical predictions, particularly from the founder of Omi Pi, who created graphics showcasing his 'wrong' takes.

He identified that critics were using bots to scrape his tweets, selectively extracting comments that appeared incorrect without considering context or sarcasm.

Theo maintained that his predictions, such as Figma's valuation trends after the failed Adobe acquisition, were ultimately proven correct despite the initial backlash.

He cited examples like Figma's valuation being down 84% from its IPO, contrary to claims that it was up 250%, and a sarcastic tweet about ChatGPT rappers misinterpreted as a serious technical opinion.

Theo argues that the rapid release of new AI models, including Grok 4.7, Opus 5.5, and GPT-6 Soul and Luna, constitutes effective 'pacing' in AI development.

He posits that the true risk in AI development is not reaching a certain intelligence level, but rather the 'takeoff' speed of recursive self-improvement, where models improve too quickly for human comprehension.

Drawing a parallel to the evolution of programming from assembly to C, he explains how increasing abstraction layers in AI can obscure the underlying mechanisms, making it harder to understand 'what and how and why' models operate.

The Jevons paradox is invoked to illustrate that as AI becomes more accessible and cost-effective, the overall volume and complexity of tasks it is applied to will expand exponentially, further challenging human understanding.

Theo explains that for certain products, a decrease in price does not necessarily lead to a significant increase in user adoption, citing refrigerators as a prime example.

He notes that almost every household in America already owns a refrigerator, and making them three times cheaper would not meaningfully increase the number of households with one, beyond a few potentially buying a second unit for their garage.

Human needs, such as hot water or refrigeration, eventually reach a plateau where further cost reductions yield diminishing returns in terms of market expansion.

This principle extends to AI, where making models more efficient or cheaper does not automatically translate to a proportional increase in new applications or users once core needs are met.

He explains that a model can possess high intelligence while simultaneously exhibiting 'stupid' or erratic behaviors, as demonstrated by GPT-6 Astra, which he describes as both the smartest and one of the stupidest models he has used.

Reinforcement Learning (RL) plays a crucial role in this refinement process by filtering out poor performance data and using good solutions from highly capable models to train smaller, less intelligent models to act less foolishly.

This approach results in models that are more reliable and cost-effective, focusing on reducing the likelihood of errors and making outputs more understandable, rather than solely pushing for higher, potentially more dangerous, capabilities.

Token usage increases 15-fold for a negligible 1% gain in accuracy when 'Max Reasoning' is enabled in Opus 5.5, the host argued.

Tests showed that 'Max Reasoning' could lead to a 15x increase in token usage while yielding only a 1% improvement in correctness, indicating a significant inefficiency.

The average runtime for tasks increased dramatically from 6 seconds to 50 seconds when using 'Max Reasoning,' severely impacting productivity and resource consumption.

Activating 'Max' mode forces the model to engage its full reasoning capacity regardless of task complexity, removing its ability to efficiently 'think less' for simpler problems, which other reasoning levels (Low, Medium, High, X-High) allow.

Theo clarifies that reasoning levels (Low, Medium, High, X-High) in AI models primarily set a capacity ceiling for how much a model *can* think, not a requirement for how deeply it *will* think.

Lower and medium settings allow models to stop reasoning once an adequate answer is found, promoting efficiency for simpler tasks.

Higher and X-High settings increase the token limit, providing more computational bandwidth for complex problems that require deeper analysis, but they do not force the model to overthink simple queries.

Max Reasoning is unique because it forces the model to utilize its full capacity, regardless of task difficulty, often leading to unnecessary resource consumption and prolonged processing times, making it generally ill-advised for most applications.

Theo observes that using different model families from various labs for code review, rather than relying on a single model, significantly enhances the detection of potential issues.

OpenAI models, in particular, are noted for their rigorous approach to code review, delving into minute details and persistently identifying concerns, even if some reported issues prove to be minor.

A benchmark comparing models in finding areas of improvement in the T3 Code codebase revealed that Astra and Grok 4.7 found eight substantiated improvements each, while Soul identified nine, though some were deemed less valid by the judging panel.

Opus 5.5 demonstrated improved performance in finding valid code issues compared to its predecessor, Opus 5, which had four supported findings out of six, whereas Opus 5.5 showed higher success and fewer unsupported findings in the judging system.