Harvard Study: AI Doesn't Boost Dev Productivity

Bain & Company estimates a $6 trillion revenue requirement within five years to justify a buildout where hundreds of billions have been spent and trillions more are expected.

Bain & Company estimates that a staggering $6 trillion in revenue will be required within the next five years to justify these infrastructure costs, even though less than 5% of ChatGPT users currently pay for the service, indicating low consumer willingness to pay for AI.

AI revenue models heavily depend on enterprise adoption rather than individual consumers, with the core assumption that AI tools will drastically enhance worker productivity.

Corporations anticipate that these productivity increases will eventually lead to the displacement of a large portion of white-collar workers, with the resulting savings from layoffs expected to offset the high consumption-based billing from AI providers.

The study tracked software engineers over six months, focusing on the practical impact of AI within real-world professional settings to assess its effectiveness.

This finding directly challenges the widely held belief that AI would significantly enhance worker output in the software development sector.

Jellyfish, a Boston-based firm that tracks code changes and task velocity via GitHub and Jira, provided the raw data for the Harvard analysis of 700,000 employees.

Jellyfish integrates with existing software development tools such as GitHub for code management and Jira for task tracking, automatically recording code changes and task velocity across its client base.

The comprehensive dataset includes over 700 enterprise customers and tracks the performance of more than 700,000 employees, comparing their productivity before and after the implementation of AI tools like GitHub Copilot, Cursor, and Claude.

Early versions of GitHub Copilot and Cursor acted as advanced autocomplete, suggesting single blocks of code while human engineers maintained complete control.

The second wave introduced AI agents, including Claude Code and Cursor's agent mode, which can take high-level tasks, plan out work, and execute across multiple files autonomously, shifting the human role from direct coding to reviewing agent-generated results.

The Harvard study utilized data from 700 companies on the Jellyfish platform, observing that agentic AI adoption began in late 2024 and quickly surged to over 90% by early 2026.

Companies were categorized into five groups based on their implementation speed, with usage rates varying significantly from under 5% to over 70% of employees after 12 months, likely reflecting diverse corporate policies and integration strategies.

While agentic AI adoption led to a substantial increase of 4,000 extra lines of code per engineer monthly and a statistically significant rise in commits and pull requests compared to standard AI assistance, this output did not translate into business value.

Crucially, the number of Jira issues and epics resolved—which serves as a proxy for business value—did not show a statistically significant increase, indicating that the higher code volume did not correspond to greater task completion.

The generation of large volumes of code, particularly low-quality output, often creates inefficiencies and increased burdens for senior engineers responsible for code reviews, negating potential productivity gains.

A senior engineer reported a 100x speedup for routine tasks due to extensive documentation training AI models, allowing quick draft generation with high accuracy, requiring only minor testing.

However, the same engineer noted that for deeper engineering tasks, AI tools did not save as much time and often produced errors, leading to more time spent in the review process; a junior engineer echoed this, stating that while AI provides 95% of the work, the final 5% demands extensive debugging.

This suggests a shift in the nature of work, where engineers transition from direct coding to primarily reviewing and correcting AI-generated output.

Junior engineers using AI can rapidly generate a large volume of code commits, but this speed creates a bottleneck at the code review stage, where senior engineers remain the sole authorities for approval due to their expertise.

Following outages attributed to AI-generated code, new rules were implemented mandating senior engineer approval for all AI-produced code, further exacerbating review bottlenecks and increasing anxiety among reviewers due to the volume and complexity of AI-generated pull requests.

While agentic AI undeniably accelerates the process of code generation, any potential productivity gains are entirely offset by the increased length and difficulty of code reviews.

The Harvard study further highlights that there was no improvement in code quality after AI implementation, with agentic AI adoption leading to a statistically significant increase in comments and change requests 12 months later, and existing AI-powered review tools proving ineffective in addressing these reliability issues within the Jellyfish dataset.

Despite predictions by figures like Anthropic CEO Dario Amade, who in May 2025 foresaw AI replacing half of entry-level jobs within a year, the Harvard study found no statistically significant impact on employment 12 months after agentic AI adoption.

Contrary to the narrative of AI-driven job displacement, companies frequently hire more workers after adopting agentic AI, and junior developer roles have not been replaced by AI, with hiring trends remaining within the margin of error.

These layoffs commenced in 2023, more than a year before agentic AI tools became widely adopted, and software engineering employment levels remain substantially higher than pre-COVID levels, suggesting that the valuation of upcoming IPOs, such as Anthropic, rests on a flawed premise of AI-induced labor displacement.