Every time a new frontier model comes out, we go through the same rituals: benchmarks get reported, leaderboards get reshuffled, and the social feeds fill up with a heady cocktail of vibes, hot takes, and everything we’ve found the new model can “one-shot.” Anyone watching from the sidelines could be forgiven for thinking that model performance is the single biggest factor driving AI transformation today.
But day to day, how much does the model really matter?
As part of my work at Jellyfish, I analyze and extract industry insights from our massive collection of software engineering workflow data, which tracks AI coding tool and agent telemetry across 300,000 developers from 1,300 companies. And when you look at what actually happens inside real engineering teams after a new model drops, the picture is a lot more complicated than the launch-day discourse would suggest.
For example: this summer, Cursor changed its defaults after SpaceX announced it was acquiring the platform. New chats started opening on its in-house Grok models, Auto mode began routing to Grok, and free users were locked into using it. Within a few months, Grok went from essentially zero Cursor requests to nearly half, while Claude, GPT, and other models fell from about half to under a quarter. This wasn’t just free users: among paying customers Grok reached nearly 40%, and a third of engineers who had been using Claude or GPT were mostly on Grok by September. A huge share of Cursor traffic switched models because the defaults changed, not because engineers went looking for something new.
So what happened?
If the model matters as much as we love to argue that it does, there should have been some meaningful change in coding output, right?
There wasn’t.
Comparing the engineers who moved the most onto Grok with colleagues at their own companies who barely touched it (more than 12,000 engineers at 222 companies), there was essentially no difference in output.
If you dig deeper, it turns out this isn’t a quirk of one particular model or vendor. When we look at the same engineers switching between major frontier models and families, none of the switches made a measurable difference.
What does matter, then?
We’ve done large-scale analyses across our data to measure which factors explain the differences in how much engineers ship, and by how much.
- Less than 1%: The tool itself (model and harness together).
- About 9%: The volume of AI that engineers use.
- About 12%: Their company, team, and culture.
- More than 75%: The individual engineer: their role, their track record, their habits, and everything else that separates today’s great AI engineers from those still figuring it out.
To be clear about the scope, this isn't saying models don't matter. Better models raise the ceiling for everyone, and the frontier is a big part of why any of this works. But among the frontier models engineers are actually choosing between today, the specific one they happen to use explains almost none of the difference in what they ship.
Put another way, for the most elite runners, there might be some small difference between two different carbon-plated racing shoes (Claude Opus 5.5 and OpenAI GPT-6 Astra), and they’re certainly a lot faster than a pair of old boots. (Opus 4 maybe?) But at the end of the day, the difference between an Olympic marathoner and an amateur dry heaving on the side of the course is their physique and training, not their footwear.
What this means for engineering leaders
For engineering leaders and practitioners this is ultimately good news, if slightly inconvenient.
The model is the most accessible lever you have, because it’s a dropdown, a router setting, or a procurement decision (and as the Cursor example showed, sometimes it’s not even your choice). But it’s also the one that moves the needle least.
The levers that matter more are slower and more challenging: getting people from occasional to deep use, building the practices and environments that let teams get good at working with AI, and investing in the people themselves.
The pace of frontier models is out of most of our hands. These other things are squarely within our control.
So the next time a model drops, don’t hesitate to try it out, and by all means share your vibes and one-shots with the world. (Otherwise I’d lose 90% of the content on my X feed.)
But if you're trying to understand why some of your teams get so much more out of AI than others, the answer is much more likely to be found in your organization than in the model picker.