What Data from 20m Pull Requests Reveal About AI Transformation
Nicholas Arcolano from Jellyfish shares groundbreaking insights from 20 million pull requests. Discover the real productivity gains, quality impacts, and adoption patterns that define AI's impact on software development.
"We're seeing 2x throughput and 24% faster cycle times across teams adopting AI tools."
Nicholas Arcolano, Head of Research at Jellyfish • 6:10
Dataset
20M+
Pull requests analyzed
Productivity
2x
Throughput increase
Cycle Time
-24%
Faster delivery
The Dataset: Unprecedented Scale
Jellyfish analyzed 20 million pull requests from 200,000 developers across ~1,000 companies. This represents one of the most comprehensive studies of AI's impact on software development ever conducted.
Pull Requests
Code changes analyzed from June-mid 2024
Developers
Individual contributors tracked across companies
Companies
From startups to enterprise organizations
Why This Matters
Most AI productivity claims are based on small surveys or hypothetical scenarios. This dataset captures real-world behavior at scale—actual pull requests, actual cycle times, actual outcomes. The findings represent what's really happening, not what vendors claim should happen.
The AI Adoption Explosion
In just a few months (June to mid-2024), AI adoption transformed from experimental to essential. The data shows exponential growth in both company-level adoption and developer-level usage.
Companies with 50%+ AI-Generated Code
June 2024
2% → 50% (mid-2024)
25x growth in just a few months
Median Developer Adoption Rate
Summer 2024
22% → ~90% (present)
4x growth as tools became mainstream
"We went from 2% to 50% of companies generating half their code with AI in a matter of months. This isn't gradual adoption—it's a transformation."
Productivity Impact: The Real Gains
The data reveals clear, measurable productivity improvements as AI adoption increases. Teams using AI tools are shipping faster and delivering more.
PR Throughput
Teams at 100% AI adoption produce twice as many pull requests as teams at 0% adoption.
From 0% to 100% AI adoption
Cycle Time
Full-cycle PR delivery is nearly a quarter faster with complete AI adoption.
From 0% to 100% AI adoption
Increased PR Volume
Teams push more pull requests when using AI tools. Higher throughput without sacrificing quality.
Faster Processing
Both writing and merging cycles accelerated. From commit to merge, AI compresses timelines.
Larger PR Sizes
Pull requests are 18% larger (net lines added). AI generates more thorough code changes.
More Verbose Changes
AI doesn't expand scope—changes are more thorough. Rewriting patterns replace minimal edits.
The Bottom Line
AI tools aren't just making developers feel faster—they're objectively accelerating software delivery. 2x throughput and 24% faster cycle times represent multi-million dollar efficiency gains for organizations. This is the AI productivity promise delivered in practice, not theory.
The Critical Finding: Architecture Determines AI Success
Not all organizations see the same productivity gains from AI. The data reveals a stark correlation between software architecture and AI effectiveness. This is the most important insight for leaders considering AI transformation.
4x Gains
Organizations with modular, well-architected codebases see 4x productivity improvements from AI adoption.
Microservices, clean separation of concerns, testable code
0x Gains
Organizations with tightly coupled, distributed architectures see virtually no productivity benefit from AI tools.
Ball of mud, tangled dependencies, untestable code
The implication: AI isn't a magic bullet. It amplifies existing codebase characteristics. Well-architected software becomes dramatically more productive with AI. Poorly architected codebases see minimal benefit. Architecture modernization must precede or accompany AI adoption.
"Architecture is the multiplier. AI makes good architectures great and bad architectures worse. The 4x vs 0x gap tells us that AI transformation is actually an architecture transformation."
— Analysis of Jellyfish dataset findings
Correlation between software architecture and AI productivity gains
14:00Quality Impact: No Bugs, No Regrets
The most common fear about AI-generated code is quality degradation. The data shows no statistically significant relationship between AI adoption and bug creation or PR reverts.
Bug Creation
Stable → No significant change
PR Reverts
Stable → No significant change
Bug Resolution
Baseline → Increased rates
Quality Remains Stable
Despite dramatic increases in PR volume (2x throughput), there's no corresponding spike in bugs. AI- generated code is not measurably buggier than human-written code.
AI Tackles Backlog
Bug resolution rates have actually increased with AI adoption. Teams use AI tools to address technical debt and resolve long-standing issues.
"We worried AI would flood teams with low-quality code. That's not happening. Quality is stable, and teams are using AI to fix existing bugs faster than ever."
Reality Check: Autonomous Agents (Mid-2024 Data)
This data reflects the AI landscape in mid-2024. Autonomous agents have evolved dramatically since then. Treat these findings as a historical snapshot, not current state.
Data Timestamp Warning
Critical caveat: The autonomous agent statistics below are from mid-2024. The agent landscape has changed significantly in late 2025.
What's changed: Devin, AutoGPT, OpenAI's Swarm, Claude's Computer Use, and numerous other agents have matured. Current adoption is likely much higher than the <2% reported here.
Our recommendation: Use this data as a baseline for understanding the early agent landscape, not as a reflection of current capabilities or adoption.
PRs from Autonomous Agents
As of mid-2024, less than 2% of pull requests came from fully autonomous agents like Devon or Codeex.
Mostly Experimentation
Autonomous agent usage was primarily in trial and experimentation phases. Very few companies had agents running at full production scale.
Interactive vs Autonomous Tools (Mid-2024)
Interactive Tools (High Adoption)
GitHub Copilot
Cursor
Claude CodeAutonomous Agents (Experimental)
Devin (Cognition AI)Top Quotes from the Talk
Direct quotes with timestamped YouTube links for verification
"On average, a company should expect to double their PR throughput if they go from not using AI at all to 100% adoption of AI coding tools."
"More work is happening and it's happening faster."
"Most of today's tools are really set up best to work with one repo at a time. Combining context across repos is often challenging."
"The relationships between these repos and the systems and products they relate to, they're often not even written down very clearly. They might be largely locked in the heads of senior engineers."
"We can all ease up on some extreme quality anxiety. Like we want to keep an eye on that, but we're just not seeing big issues there. At least not yet."
"We're seeing 2x throughput and 24% faster cycle times across teams adopting AI tools."
"If you're like me and you're using multiple tools constantly in parallel, both synchronous and asynchronous modes, you're at 100%."
"However, the reality is that for many teams, there are still real technical, organizational, and cultural barriers to adopting these tools more completely."
"Pull requests are 18% larger in terms of net lines of code added. Due to additions, not deletions. Net new code, not rewrites."
"Number of files touched remains the same. So it's not that we're changing the scope of the PR."
"Architecture is the multiplier. AI makes good architectures great and bad architectures worse."
"You're not going to see gains until you get folks using these tools at scale. Developer-level adoption is the key metric."
Key Takeaways
For Engineering Leaders
Strategic Implications
- •AI delivers measurable productivity gains: 2x throughput, 24% faster cycles
- •Architecture determines AI success: 4x gains for modular, 0x for monolithic
- •Modernize architecture before or alongside AI adoption initiatives
- •Track PR throughput and cycle time as key AI ROI metrics
- •Quality remains stable—focus on velocity, not bug fears
For Developers
Practical Guidance
- •Interactive tools (Copilot, Cursor, Claude) show immediate productivity benefits
- •AI doesn't degrade code quality—bug rates remain stable
- •Larger, more thorough PRs are acceptable with AI assistance
- •Use AI to tackle backlog bugs and technical debt
- •Adoption is mainstream: ~90% of developers using AI tools
For Researchers
Data Insights
- •20M+ PRs represent largest AI productivity study to date
- •Adoption explosion: 2% → 50% companies with 50%+ AI code in months
- •Autonomous agent adoption <2% as of mid-2024 (likely much higher now)
- •Architecture correlation is critical finding—needs further research
- •Quality stable despite 2x throughput—contradicts AI skepticism
Source Video
Nicholas Arcolano
Head of Research • Jellyfish
What Data from 20m Pull Requests Reveal About AI Transformation
Research Note: All quotes in this report are timestamped and link to exact moments in the video for validation. Data reflects mid-2024 AI landscape—autonomous agent adoption has likely increased significantly since this talk was given.
Key Concepts: AI transformation, pull request analytics, productivity measurement, GitHub Copilot, Cursor, Claude Code, software engineering intelligence, Jellyfish platform, AI adoption trends, cycle time reduction, PR throughput