Opus 5.5, GPT-6 Sol, and Luna Surprise in Live Review

These models are positioned as daily drivers for coding and knowledge work, with an emphasis on performance and cost-effectiveness.

This expansion addresses the increasing sophistication and agentic capabilities of newer models, utilizing a blind taste test and an LLM judge despite acknowledged subjectivity.

New AI models prioritize not only cost reduction but also improvements in output generation, token usage, and overall speed.

Opus 5.5, for instance, is currently twice as expensive as GPT-6 Soul, yet significant focus remains on optimizing cached inputs to prevent unexpected expenses from poor cache management.

Designed to block or downgrade requests deemed risky, this model incorporates built-in cyber and bio security constraints that follow extensive external evaluation.

The model actively blocks or downgrades requests deemed risky in cyber or bio tasks, reflecting its conservative nature.

This conservatism, however, can lead to the model refusing certain tasks, a behavior noted by the host.

GPT models, especially GPT-6 Sol, demonstrate significantly faster performance compared to Claude models, according to the reviewer's experience.

While Claude 3.5's reduced narration can create uncertainty about its processing status, GPT models maintain speed while providing an appropriate level of narration, contrasting with Claude's 'high effort' background processing that contributes to perceived slowness.

New categories such as front-end and back-end coding, agentic workflows, and creative features have been added, with models from OpenAI, Anthropic, Grock, and Muse under evaluation, anticipating varied preferences based on specific task requirements like agent personality.

During blind tests where names like Model B, C, E, and G ensure impartiality, Model B achieved the top score by producing readable emails complete with calendar invites.

Model C, identified as Grock, received criticism for its low token usage, which led to confusing and low-context responses.

The reviewer also expressed annoyance with models that excessively use em dashes, calling them 'slop'.

The reviewer subjectively scored AI models on writing tasks, giving a model a three for short but understandable output, and another a four for correctly omitting unnecessary information.

Luna achieved a top score of five by meticulously adhering to explicit email writing rules.

This evaluation process mirrored a real-world scenario where a recipient reviews task output for effectiveness and clarity.

The assessment critiqued model outputs on color usage, empty state handling, and layout, with the reviewer conducting blind taste tests to uncover inherent design patterns and weaknesses rather than relying on general one-shot prompts.

Consumer-focused designs frequently lean on forest green and sage palettes, yet these choices often lack the professional polish required for SaaS product standards.

Many consumer-focused designs were labeled 'slop' due to poor text weight and redundant layout choices, often defaulting to uninspired templates reminiscent of Claude Artifacts, indicating a lack of consistent and creative design.

Models show improved proficiency in generating functional SVG illustrations, a notable advancement in their capabilities.

However, the reviewer identified a recurring and frustrating design pattern where models consistently place circles in corners, while Model H distinguished itself with superior SVG precision and overall visual appeal.

App designs for development tools often struggle with excessive information density, hindering usability and clarity.

Models receiving higher ratings were commended for producing cleaner, simpler, and more readable interfaces, with bonus points awarded for creative designs that deviated from the conventional dark mode aesthetic.

While each model successfully audited code and built backend features to specification, the depth of their output varied significantly, prompting the speaker to employ an 'LLM as a judge' to verify technical readability.

A significant variation was observed in how models presented their output, with messages ranging from extremely concise to highly detailed, impacting readability for PR descriptions and technical specifications.

The 'LLM as a judge' methodology will be employed to verify the correctness of the code, underscoring the importance of clear presentation for effective communication.

The quality of information delivery directly impacts how easily backend code and specifications can be understood and audited.

Model B and Model E were identified as favorites for their capacity to categorize information into top problems and conflicts during documentation tasks.

The evaluation criteria favored models capable of categorizing information effectively into key problems and potential conflicts, enhancing readability and utility.

Models were tasked with generating specific SVGs, including a document, a microphone, and a bug, with varying results in quality.

Model H received the highest rating for its superior detailed rendering, notably producing a microphone with proper shadows that distinctly resembled a microphone rather than a cactus.

Initial predictions suggest Claude/Opus 5.5 might excel in front-end tasks, while Soul could be preferred for writing, with the reviewer also referencing her 'Barbie Bench,' a personal benchmark involving a 3D Barbie fashion video game.

GPT-6 Astra and GPT-6 Soul are favored for personal preference and character SVG work, while Opus 5.5 garnered the most high scores (4s and 5s) across the broadest range of tasks.

Opus 5.5 is specifically noted as most effective for agentic tasks and long-running workflows, whereas GPT-6 models are lauded for their readability and simplicity in Product Requirements Document (PRD) generation, leading to Fable being dropped from the host's workflow due to lower performance.

Opus 5 and 5.5 continue to be criticized for being overly verbose and chatty, indicating persistent issues with conciseness despite efforts to reduce verbosity.

In contrast, Astra and Soul demonstrated superior performance in generating character SVGs, while Claude Opus maintained its preference for B2B renewal tasks, revealing a discrepancy between the host's preferences and the rankings produced by LLM judges.