Why AI-Generated Writing Continues to Fall Short: The 'Slop' Phenomenon and ToneBench Benchmark

Louis-François Bouchard of Towards AI describes 'slop' as a failure mode in AI-generated writing that makes content feel artificial. He notes that this issue persists and even scales with the improvement of language models, such as GPT 6.1, rather than diminishing.

Slop is characterized by six main traits: overused words (like 'delve'), excessive generality, content that says nothing substantial, repetition, hallucination, and either an incorrect tone or a lack of any personal tone, making it sound like everyone and no one at the same time.

Louis-François Bouchard elaborated on the six primary characteristics that define 'slop' in AI-generated writing. First, it frequently uses words that are common for LLMs but not in human conversation, such as 'delve.' Second, the text tends to be overly general, often discussing 'most teams' or 'most projects' without providing specific details. Third, the writing often says nothing meaningful, merely stringing together pleasant-sounding words.

Fourth, slop repeatedly conveys empty information. Fifth, it hallucinates, meaning the content cannot be fully trusted. Finally, and most problematic for Bouchard, sloppy writing usually has an inappropriate tone or no distinct tone at all, resulting in a generic voice that fails to reflect the user's individuality.

To address the challenge of evaluating AI writing quality beyond simple word counts, Louis-François Bouchard's team developed and released a new benchmark called ToneBench. The benchmark's primary goal is to measure the degree of 'sloppiness' in generated text, specifically focusing on creative writing and a model's ability to adapt to a unique voice.

Bouchard clarified that ToneBench is not a reimagining of EQbench's slop score but rather a distinct tool for quantifying the subjective qualities of AI-generated prose. The team plans to detail the benchmark's methodology, measurement criteria, and the costs involved in its development and execution.

The development of ToneBench stemmed from a practical need within Louis-François Bouchard's company, Towards AI, which produces extensive educational content including hundreds of videos, thousands of articles, and books. As a small team heavily reliant on AI for content creation, the constant review and correction of 'slop' became a significant and annoying burden.

The demand for such a benchmark was also reinforced by strong community engagement on platforms like X (formerly Twitter), where users expressed a clear need for a creative writing benchmark that could help ensure higher quality AI-generated content.

Louis-François Bouchard emphasized his team's expertise in AI model evaluation, having built evaluation pipelines for clients and possessing a staff of 15 AI engineers, creative writers, and editors. This background, he states, makes them well-suited to develop the ToneBench benchmark.

Bouchard defines successful AI writing, specifically 'writing like me,' as content he could read aloud without cringing at words or phrases he would not personally use. The benchmark aims to assess if a model can generate video scripts that authentically reflect his unique voice and style.

The ToneBench benchmark utilizes extensive and detailed prompts for each model. These prompts include in-depth information about Louis-François Bouchard's speaking and writing style, numerous examples from his previous video content, and guidance on general storytelling principles and the use of active sentences.

Each prompt also outlines the specific task for the model to generate one video script. The benchmark consists of 10 distinct scripts, some previously published and others unpublished, which are kept hidden from the models to prevent contamination and ensure novel generation.

For each of the 10 scripts in ToneBench, every model generates five distinct drafts. This approach of generating multiple drafts is a deliberate choice to measure intramodel variability, as some models, like GPT-4.7, can produce outlier drafts that perform significantly better than their average. Louis-François Bouchard noted that the worse a model performs, the wider its variance tends to be.

After generation, four different LLM judges analyze each of these drafts. The decision to use five drafts was chosen as a cost-effective middle ground to effectively capture a model's range of output without incurring excessive expenses in the evaluation process.

ToneBench evaluates AI-generated writing across nine detailed metrics, which Louis-François Bouchard condenses into five main aspects. These aspects include tone, assessing if the writing aligns with his voice; pure writing quality, focusing on elements like hooks, emotional impact, and transitions; and instruction following, checking for adherence to specified length, format, and image placement.

Additionally, the benchmark measures research accuracy, to detect hallucinations and ensure correct use of provided research briefs, and the presence of 'slop,' using both EQbench's slop score and proprietary measurements.

Each LLM judge in the ToneBench benchmark receives an extensive prompt, approximately 14,000 tokens long. This prompt includes around a thousand words of examples, instructions, and directions for each metric, alongside the script to be analyzed, Louis-François Bouchard's own script for guidance, and the project brief and research material.

The judges are specifically instructed to analyze only writing quality, remaining blind to the length of the output and the originating model to prevent bias. The model's identity is attached later using code. For subjective metrics like tone and voice, judges first categorize the text into one of six predefined categories before assigning a rating, a method found to be more effective than binary or direct percentage scoring for qualitative tasks.

The sheer volume of content generated by the ToneBench benchmark necessitates an automated evaluation process. With 120 models each producing five drafts for 10 distinct tasks, the number of scripts quickly escalates into the thousands, making manual human review of every single output impractical and impossible for the team.

Louis-François Bouchard highlighted a significant issue with using LLMs as judges: they often exhibit strong biases, particularly favoring models from their own families. He noted that OpenAI models, for instance, show a clear preference for their own family of models. Bouchard found it 'quite funny' that Entropic and OpenAI models appear to be in a 'war,' suggesting they may have undergone post-training that encourages a negative bias towards each other's outputs.

To ensure objective evaluation, especially in creative tasks, Bouchard advises users to employ different model families when using LLMs as judges. The team also conducted 1,000 human A/B comparisons of generated script chunks to validate the LLM judge results and plans to seek community assistance for future human evaluations due to the impracticality of extensive manual review.

Building a high-quality AI writing benchmark like ToneBench is a resource-intensive endeavor. Louis-François Bouchard stated that the initial development involved approximately 500 hours of work, dedicated to crafting 10 scripts, defining all rubrics, finding examples, and the challenging task of quantifying writing quality.

Furthermore, iterating on the benchmark's parameters proves costly: a single change in a rubric can incur over $2,000, while modifying just one line in an input prompt can cost more than $4,000. These expenses highlight the significant financial investment and careful planning required for such experimental changes.

Louis-François Bouchard emphasized that effective AI writing is not about achieving a perfect 'one-shot' output from a model, but rather about finding a model that is the 'least annoying' to use in reaching the desired outcome. After four months of testing creative writing models, he concluded that immediate perfection in AI-generated creative content is not currently attainable.

Instead, the focus should be on optimizing the workflow: using extremely long and detailed prompts with examples, employing a strong model for one or a few review passes to prevent 'slop,' merging sections from multiple cheap drafts, and always performing a final human edit. He also noted that lightweight models like GLM 5.3 Flash offer significant cost-effectiveness while maintaining competitive performance compared to heavier, more expensive options.

Louis-François Bouchard shared a key lesson from four months of testing creative writing models: the goal isn't to find a model that generates a perfect 'one-shot' output. Instead, users should select the model that is the 'least annoying' to work with to achieve their desired results.

This approach acknowledges that no current model delivers immediate perfection in creative generation. The emphasis is on streamlining the overall workflow and reducing friction in the iterative process of refining AI-generated content.

In terms of cost-effectiveness at scale, Louis-François Bouchard highlighted GLM 5.3 Flash as an exceptionally good and cheap option. It is 50 times cheaper than the top-performing model, Opus 5.5 Extra High.

He also noted that Opus 5.5, while more expensive than GLM 5.3 Flash, is still 2.5 times cheaper than 'max thinking' models while delivering comparable results, suggesting that these highest-tier models are often unnecessary for typical writing tasks.

To achieve the best results with AI writing models, Louis-François Bouchard strongly advocates for using extremely long and detailed prompts. He stresses that prompts can never be too long, and users should include every conceivable positive and negative example, including entire previous works like newsletters, to guide the model effectively.

Another key practice is to use a high-thinking model as a judge for one to three review passes. This prevents the initial model from falling into 'slop' and provides crucial feedback for improvement. Ultimately, a final human review remains indispensable for all AI-generated content.

For large-scale content production, Louis-François Bouchard recommends a strategic approach that leverages both cost-effective and high-reasoning models. He suggests generating multiple drafts, perhaps 10, for a specific task using a cheaper model like GLM 5.3 Flash.

Following this, a 'smarter,' high-thinking judge model, such as Opus, is employed to select and merge the best parts from these various drafts. This multi-draft merge strategy is projected to be significantly less expensive than a single generation from a premium model like Opus 5.5 and is expected to yield superior results.