Marques Brownlee Criticizes YouTube's New A/B Testing

Marques Brownlee, a prominent tech reviewer, argues that YouTube dedicates too much effort to mimicking other platforms instead of reinforcing its unique strengths.

He specifically points to the recently unveiled video A/B testing feature as the most perilous and potentially detrimental concept introduced by the platform to date.

Creators have utilized A/B testing for thumbnails and titles since last year, a practice Brownlee initially resisted but eventually came to accept as a tool for optimizing content.

However, he draws a distinct line, stating that applying the same A/B testing methodology to the actual video content itself is a step too far for him, raising significant concerns.

YouTube's new A/B testing tool lets creators upload various versions of a video, including shorts or long-form content, to compare which version best retains audience attention.

Upon completion of the test, the system furnishes three distinct retention graphs, aiding creators in identifying the most effective video cut.

Brownlee questioned how viewers would navigate comments and timestamps if different video versions are in circulation, citing potential confusion for users referencing specific moments.

He also expressed concern about consistency, highlighting that sharing a link could lead the recipient to a different video version than the sender originally viewed, disrupting the shared experience.

A significant unresolved issue, according to Brownlee, is the lack of clarity regarding the permissible extent of deviation between the video versions submitted for testing.

Brownlee's experience with thumbnail testing over the past year reveals that minor alterations, such as slight text size adjustments or arrow placements, rarely produce statistically significant results.

He notes that meaningful performance insights typically emerge only when the tested concepts are fundamentally distinct, leading him to question the utility of applying a 'retention maxing' strategy to the actual video body for learning.

He wonders if applying this 'retention maxing' strategy to the actual video body will provide meaningful learning for creators.

Brownlee questioned the permissible difference in video versions, citing YouTube's own examples that show variations like 17 versus 19 minutes in length.

He also expressed strong concerns that the A/B testing mechanism would inevitably break existing timestamp functionalities for viewers.

To address these implementation challenges, Brownlee engaged in a discussion with YouTube engineers to gain further clarity on their plans.

Behind the scenes, YouTube has operated a restricted tool for replacing video files for several years to address specific factual corrections.

This tool is currently reserved for precise factual corrections, such as fixing an error in an Apple commercial.

A key restriction is that any replacement video must be almost identical in length to the original file, with only the correction itself accounting for any variance.

To ensure A/B testing precision, YouTube will require videos to meet a specific similarity threshold starting early next year, a process Marques claims may suffer from reliability issues due to existing limitations in transcript-only analysis.

Only videos that achieve a sufficiently high similarity score, as determined by this model, will be eligible to run as A/B tests.

Brownlee expressed skepticism about the model's efficacy, drawing a comparison to Gemini's current limitations in analyzing video content beyond mere transcripts.

A/B testing will be restricted to the initial launch of a video, preventing its use on existing content.

Both uploaders and viewers will remain unaware of which video version is being tested, which could lead to significant confusion regarding comments and content references.

Brownlee questions the system's ability to detect substantial content differences, such as altered conclusions or reordered segments, further highlighting potential ambiguities in the testing process.

Marques Brownlee contends that YouTube's essence lies in an implicit agreement among viewers to share a common experience, all watching the same video.

He argues that enabling creators to upload multiple versions of the same video simultaneously fundamentally disrupts this shared cultural phenomenon, even if minor packaging differences existed before.

Brownlee criticizes YouTube's ongoing tendency to adopt features from competing platforms, which he believes erodes the platform's distinct community identity.

YouTube previously removed the public display of dislike counts, a metric that creators found uniquely useful for gauging audience sentiment.

The platform has also revised its view counting methodology, now registering a view as soon as the first frame of a video loads, which is a more liberal approach.

Brownlee contrasts this with TikTok's view-counting method, implying that YouTube's earlier, more stringent view measurement made each individual view a more valuable indicator of engagement.

Brownlee advises that the vast majority of creators (99.9%) should avoid using the new A/B testing feature for videos.

He argues that over-optimization for retention often leads to the removal of unique, fun, and characterful elements that define a creator's style.

Brownlee emphasizes that success on YouTube is not solely defined by mass appeal, noting that niche content contributes significantly to the platform's overall value and diversity, and constant optimization could strip away essential creative storytelling elements.

Brownlee suggests that creators' time is better spent producing and publishing a greater volume of videos rather than investing excessive effort into endlessly refining a single project for marginal gains in views or average view duration.

He advocates for creators to maintain their original creative vision instead of relying on automated A/B tests to dictate content choices.

The genuine creative process, according to Brownlee, involves completing a video, drawing lessons from both its successes and failures, and then applying those insights to subsequent projects.