Better AI Models Won't Fix Your Engineering Team: Focus on People, Not Just Prompts

Nicholas Arcolano, Head of AI and Research at Jellyfish, introduced his talk by asserting that if teams are not achieving desired outcomes with AI, the problem often lies with the team itself rather than the AI model being used.

He utilized a golf analogy, comparing the pursuit of better AI models to buying better golf clubs, while emphasizing the importance of fixing fundamental skills, which he equates to human factors in engineering productivity.

Arcolano acknowledged that better equipment does matter, citing data from the PGA tour. The average driving distance for PGA players has increased by approximately 20% since 1980, from 257 yards.

A notable jump of about 15 yards occurred around 2000, coinciding with the solid core ball era, which was attributed to technological advancements rather than a sudden improvement in player skill over a three-year period. This illustrates how technology can undeniably impact performance, setting up a comparison for AI models.

In a parallel to golf equipment, Arcolano conceded that AI models have indeed improved substantially. He referenced Artificial Analysis benchmark charts, which consistently show models getting better.

He highlighted that capabilities that were impossible just a year ago are now achievable, underscoring that the models themselves are a crucial factor in what can be accomplished.

Jellyfish collects telemetry data from AI coding tools and agents, combined with engineering workflow and outcome data, covering hundreds of thousands of developers across approximately 1,300 companies. This extensive dataset provides insights into AI's real-world impact.

Around the time of SpaceX's acquisition of Cursor, there was a rapid and significant increase in Grok usage, faster than typical user preference shifts. This spike was primarily driven by Cursor's UI changes, which promoted Grok and altered how its auto mode functioned, creating a 'natural experiment' for analysis.

Despite this major shift to Grok, a statistical analysis revealed no correlation between increased Grok usage and changes in merged Pull Requests (PRs) per engineer. The output remained largely unchanged, indicating that the model switch did not statistically impact productivity.

The analysis focuses on a curated subset of approximately 50,000 engineers from 447 companies, specifically examining data from this year involving Cursor and Claude Code users.

The key output metric used for this study is 'merged PRs per engineer per week.' The framework for understanding productivity compares the AI 'tool' (harness and model combined) to golf clubs, and 'skill use' to the golfer's swing.

Arcolano likened harness customization and other AI engineering 'tricks of the trade' to knowing how to escape a bunker in golf. He noted that these are not innate abilities but skills that engineers must actively learn to optimize their use of AI tools.

The analysis mapped four main factors influencing PR throughput: the tool itself, skill use, token usage, and the company/team environment, with the individual engineer accounting for the largest share of remaining variability. These factors were statistically analyzed, holding others constant.

The tool choice, encompassing the AI model and its harness, explained less than 1% of PR throughput change. Skill use accounted for about 3%, while token usage contributed approximately 9%. The company and team environment explained about 12% of the variation, leaving the individual engineer as the most significant factor, explaining the majority of the remaining throughput.

Arcolano presented data comparing engineers against themselves when they switched AI models or harnesses. Examples included changing from Opus to Sonnet, switching to ChatGPT, moving from Cursor to Claude Code, or transitioning between autocomplete and agentic modes.

Across these scenarios, the statistical impact on PR throughput was described as a 'nothingburger,' indicating that such switches did not lead to any meaningful changes in output.

While model choice has minimal impact, skill use significantly affects output, far more than selecting a different model. However, data from September indicates that almost half of engineer-weeks had virtually no skill use.

Only 12% of weeks demonstrated skill usage levels considered consistent with modern AI engineering practices. For engineers engaging in heavy skill use (defined as 30 or more runs per week), there was a substantial 30% increase in PR throughput, even when other factors were kept constant. This effect is considerably larger than any observed impact from changing AI model families.

Based on data from approximately 500,000 developer weeks, more tokens generally lead to higher throughput, but with diminishing returns. The analysis, previously presented at the AI Engineer World's Fair, shows a saturating curve, which Arcolano likened to the rocket equation.

Engineers in the top 10% for token consumption roughly double their output compared to the median, but this increased output comes at a cost of about 10 times the tokens. This highlights that while increased usage can improve productivity, the efficiency gains decrease at higher consumption levels.

The organizational environment, encompassing company and team factors, accounts for a substantial portion of the variability in AI-driven engineering output. This factor is broken down into approximately one-third related to the team and two-thirds attributed to broader company aspects.

These factors include elements such as the code base, review processes, and organizational culture. Even when AI token consumption is held constant, there is a 2x productivity spread between the top 10% and bottom 10% of companies, underscoring the profound impact of the work environment. Arcolano emphasized that these are areas leaders can directly control.

Individual engineers account for roughly three-quarters of the statistical model explaining throughput, making their contribution the most significant factor. The productivity spread among individual engineers is even wider than that among companies, showing a 3.5x difference between the bottom 10% and top 10% of performers.

Longitudinal data from Jellyfish, spanning 10 years, indicates that AI has not fundamentally reshuffled the hierarchy of top performers. Engineers who were top performers before AI largely remain top performers after its adoption. This challenges the narrative that AI is leveling the playing field.

Analysis showed that the percentage of top-quartile engineers cycling out of that rank every six months slightly decreased from 44% to 41% after the integration of AI, suggesting a slight stabilization rather than a disruption of performance hierarchies.

Arcolano advised that while AI tools and foundation models garner significant attention, engineering leaders have limited direct control over their improvement. Instead, leaders should concentrate on factors within their direct influence: the people, the organization, and the working environment.

Unlike a golfer who cannot change the course, engineering leaders possess the agency to modify their 'course'—the team and company environment—to enhance outcomes. This includes fostering skills, optimizing review processes, and cultivating a productive culture.

He concluded by reiterating that if desired outcomes are not being met, the root cause is more likely to be found in team and environmental factors than in the AI model itself, urging leaders to focus on what they can control.