Rendered at 08:11:51 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ianbutler 55 minutes ago [-]
2025 is such old news that this just isn't relevant.
METR already redid the study at a later date and now finds a likely 18% speedup
"For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post)
Their followup study essentially says the followup study is possibly broken because developers will now not participate in some of the non-AI tasks and because the study pays less.
I would not, at all, suggest that this second study corrects or debunks the first.
Instead what it shows (if anything, i.e. if you can even put aside the regrettable choice to change the payment level, which affects applicant recruitment) is that the mindset shift has already happened: developers now don’t want to attempt some tasks without AI.
What that tells you is not with any confidence that they are faster, but that we are possibly beyond the point that this can be meaningfully measured. AI could still be making developers slower, but developers aren’t going to be willing or perhaps able to help you find out.
Basically the job is different now.
What this does for me, perhaps, is vindicate my feelings. I can do agentic coding; I have learned the principles and some tools and I could learn more. But if this study is really reflective of how other developers feel now, I am done.
torginus 38 minutes ago [-]
Either way, it's not a dramatic improvement. Thankfully I work in an environment where with little bureaucracy so my time is actually spent doing technical work.
I do think AI has been a huge boon to productivity in many ways, but looking at feature timelines, I think it's pretty clear the 'critical shortest path' of key features hasn't been sped up by that much.
js8 39 minutes ago [-]
They also say "Wider adoption of AI has made it more difficult to measure task-level productivity"
I think there is a simple reason for that. If you automate something, you make the measureable/predictable thing faster. So the hard to measure/predict part of the job will take more share of the time, and overall difficulty to measure/predict goes up.
I think this is what happened with Agile Scrum - as developers became more productive (for unrelated reasons, two main sources of SW developer productivity before AI were compilers and open source), the bureacracy (amount of meetings) increased, because the ratio of hard to measure vs easy to measure went up. Bureacracy is hard to measure, so it went up (as a share of work). I expect this only getting worse with more automation, such as AI. So I predict an increase in share of bureacracy compared to pre-AI world.
Either way, IMHO main point is automation has the opposite effect on human job predictability, it lowers it. Tasks we can easily automate are those that are easy to predict.
nikau 36 minutes ago [-]
I've held this stance on agile for a long time - it coincided with mainstream adoption of ssds, windows with memory protection and google search - all of which sped delivery despite agile, not because of agile.
verzali 40 minutes ago [-]
That post literally says the results are unreliable...
lars512 35 minutes ago [-]
...and in particular it says that one of the reasons is that developers are refusing to participate in the non-AI branch, and when they do, changing what tasks they select to those where AI would be less useful.
Overall this suggests to them that the current speedup is likely greater than what the study could measure.
dofm 19 minutes ago [-]
It might suggest that but they can’t back it up because the study is broken and perhaps forever unrepeatable.
Like, what people are saying is, “That old study was wrong! They did a new broken study that overturned it!”
krige 42 minutes ago [-]
Even not touching the laughable sample size for both studies - almost halved sample size between 2025 and 2026? Sounds like a massive selection bias, and not in the way they're implying.
markbao 41 minutes ago [-]
And to be specific, the METR study was using the Cursor harness with Claude Sonnet 3.5/3.7, along with other models of that era of the participant’s choosing.
Which is ancient at this point, and half a year older than the November 2025 inflection point when agentic coding got really good.
The original article is from August 2025, and the overall message to not trust ‘how it feels’ and rather measure outcomes seems right to me despite the outdated figures. On my team at least, we are seeing a noticeable inflection in work shipped with AI according to Weave.
loveparade 56 minutes ago [-]
These studies are meaningless because speedup is heavily dependent on the kind of work you're doing. No doubt that you can do mechanical refactors 100x faster with AI, and also no doubt that using AI will be slower for tasks where it's less about writing code and more about context/world knowledge or building understanding. Averaging across these tasks doesn't make sense because everyone's work consists of a different distribution of tasks.
A frontend dev doing tailwind integration for his day job is gonna see very different speedups than someone working in a niche scientific codebase. Taking the average makes about as much sense as taking the average of the speedup from calculators for a mathematician, a farmer, and an elementary school student.
rob74 33 minutes ago [-]
To add to that, the only way to reliably measure speedup would be to give the same developer the same task twice, first without AI, then with AI, and the developer should have no previous knowledge (or the same level of previous knowledge) about the task each time he starts - which is inherently impossible. I didn't read the study, but from the article it looks like they compared the actual speed to prior estimates, and we all know how reliable those are?
dakolli 39 minutes ago [-]
I have found llms to be utterly useless for frontend (tailwind included).
That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off.
I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "world", customer and product knowledge is required to design and implement great UX/UI?
I promise you are not going to be able to translate all this internalized understanding to an LLM and have it do your "tailwind integration" It actually sucks at all frontend outside of the 3 types of page layouts it understand.. Shitty landing pages, generic dashboards and shitty blog layouts.
Ya'll yearn for slop though so maybe everything will just become shit anyways.
loveparade 37 minutes ago [-]
Fair point, I was more trying to make a statement about the amount of training data available, not the "difficulty" of the task. I just used Tailwind as an example because it is so ubiquitous with so much training data for LLMs to learn from, while any niche application doesn't have that.
31 minutes ago [-]
shaky-carrousel 1 hours ago [-]
There was a study that people using the keyboard instead of the mouse felt they were working faster but in fact they were working slower. A perception thing. Users were more engaged when using a keyboard.
shawabawa3 59 minutes ago [-]
I'm convinced this is what causes people to feel productive with vim
100ms 53 minutes ago [-]
Triggered by both of these comments.. interaction mode dictates a style of thinking. I have to use a mouse, I'm forced to use my eyes, which also means I probably have to use a massive screen. I have to pay attention to some hyperactive Intellisense-like feature, I'm forced to remove my attention from the problem.
It's like saying you're convinced people reporting they feel more productive in a mauve-coloured room are liars, or those that drive automatic vs manual. Maybe they just find muave a restful colour?
dakolli 37 minutes ago [-]
people that use vim motions/shortcuts/keyboard workflows are more productive, this is undeniable..
Shitty-kitty 14 minutes ago [-]
Vim makes some slow and incredibly tedious tasks, fast and efficient. Having said that, all those key-presses to switch modes do add up.
Rekindle8090 22 minutes ago [-]
[dead]
michaelt 27 minutes ago [-]
One thing I've noticed with generative AI is it's now easier than ever to write more lines of code.
Before, a backend guy asked to add an intranet page would make an austere page -bare html with barely any styling or javascript. Today, the same guy given the same task can turn in something with styling, javascript, internationalisation, interactive form validation, progress spinner, minification build stage, linting, maybe even automated browser tests.
And I have to code review it. Now the bottleneck of writing the code has been removed, I now find code review is the bottleneck - and a bottleneck facing much higher flow must either let more through, or start applying back pressure.
Sometimes I think an evil genie granted my wish for better tested code by trying to drown me in it.
jdkoeck 50 minutes ago [-]
> The honest counter, and it matters here more than usual. This is most likely the dip in a J-curve, not the destination.
Oh, the irony of this post being AI-generated.
titanomachy 34 minutes ago [-]
I don’t understand how these things still write so annoyingly. Eliminating just a handful of tells would make a big difference.
xg15 31 minutes ago [-]
> AI speeds up typing, which was never the bottleneck for an expert in a codebase they already know.
For me as a dev, that's not the whole truth. Where I've found actual value in AI (and I think were some of that "perceived speedup" is coming from) is looking up things.
Unless you know the codebase and used libraries extremely well, you will have to do lots of "micro-lookups" during coding, where you have to find the specific APIs or library functions for your problem, then figure out how exactly you have to call them, how to handle the result, etc. That's lots of "research" work interleaved with actually writing the code.
AIs seem to be good enough to have a lot of that knowledge already baked into their weights, at least for popular platforms, so if you prompt it something, you can skip all that low-level lookup work or at least defer it until code review. Even during review, it's easier, because you don't have to come up with the appropriate library function from scratch, you only have to verify that the ones the AI used make sense and are used correctly.
onion2k 53 minutes ago [-]
Generation got cheap. Verification got expensive.
That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing.
Alternatively, rather than trusting AI to do the validation, we could follow the vibe-coder approach by skipping the validation entirely, and trust that the generation stage is good enough not to need it. Historically that's come with some small downsides, like the code being a broken mess of security holes, but with time AI might fix that.
hnlmorg 51 minutes ago [-]
I wouldn’t even trust experienced developers to merge code without peer review.
onion2k 37 minutes ago [-]
I would, but that's mostly because I don't trust PRs to catch real problems. Someone reviewing the changes in a codebase is never going to spot an architectural or code design issue, and those are the real ones you need to care about. In my experience 95% of everything that's caught by a human PR review could have been caught by a linter or a formatter before the PR was opened.
If you trust your team to care about quality then PRs aren't necessary, and if you don't then why are you trusting them to catching problems in PR reviews?
roenxi 40 minutes ago [-]
It is getting to the point where I wouldn't trust experienced developers to merge code without AI review. The latest generation of models are getting pretty good.
alfalfasprout 51 minutes ago [-]
small downsides like security holes? Those aren't small. Neither is creating a codebase that's an inextensible mess that even LLMs begin to struggle with.
The reality is making good decisions and thinking about approaches take time. AI can absolutely make us faster at it but it's not magic and these speedups come with effort.
onion2k 31 minutes ago [-]
small downsides like security holes? Those aren't small
I'm British. I've been taught to turn understatement into an art.
sigmoid10 40 minutes ago [-]
If you linked to the actual source of the study [1] instead of a random blog only talks about the result, you would see the big banner that the authors put there noting that the study is horribly outdated. Current models do make developers faster.
As I read the blog post, I thought that it was released today. Maybe point out that it is almost a year old. It feels like it is manipulating HN users.
And this is coming from an AI sceptic.
wewewedxfgdf 54 minutes ago [-]
My two bosses are anti-AI.
Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors.
People will always create a world narrative that matches what they already believe.
Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it reinforces their world view.
Almondsetat 49 minutes ago [-]
>feel
Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist
Shitty-kitty 7 minutes ago [-]
Problem is, all the old metrics are now obsolete and nobody knows how to measure productivity anymore.
46 minutes ago [-]
dymk 44 minutes ago [-]
Performance review at FAANGs has always been vibes and soft skills.
hparadiz 41 minutes ago [-]
Shipped projects don't lie.
Almondsetat 26 minutes ago [-]
And? That's not the point of discussion
41 minutes ago [-]
croes 51 minutes ago [-]
But this isn’t just a narrative but a study. Limited but a study
jdkoeck 49 minutes ago [-]
A study from 2025. Might as well be from five years ago.
mschuetz 43 minutes ago [-]
Having participated in many studies, I lost much faith in studies. You could study the same thing, and get opposite results depending on how you build the study, which people participate (friends and colleagues will be reluctant to speak against your results), and the bias with which you set everything up. Also, many tasks require learning the tools, and some tools will start to be more productive with expertise than others.
49 minutes ago [-]
vachina 50 minutes ago [-]
It will feel slower because I finish my task in 1/20th the time and spend the remaining time browsing HN.
The actual study with the data, minus the "I was right all along" commentary
raincole 45 minutes ago [-]
Yeah, when people who are not familiar with AI and use Cursor with Sonnet 3.7 they are only 19% slower. In retrospect that research was very bullish for AI.
make_it_sure 53 minutes ago [-]
why is this on the front page. It's an old debunked study lol, not relevant at all today. Read more than just the title
miika 43 minutes ago [-]
Well.. feelings have never been a good way to measure quantities.
larodi 26 minutes ago [-]
So are you trying to imply that I've somehow accidentally stumbled upon more than 100k lines of new high-perf working code, done in less than 6 months, which is like not 20%, but 200% my actual output, and this code, already generating revenue for me and my employer, is something of the ordinary, and actually I can do 20% better typing it manually?
ROFL sorry
44 minutes ago [-]
bezier-curve 43 minutes ago [-]
One thing I see missing in a lot of these discussions is whether or not the metric is solely based on speed. I think AI just allows you to look at your code in different ways and provides more chances to catch mistakes. I am definitely slower with AI assistance, but that is because I use it to increase the quality of my work.
DonHopkins 44 minutes ago [-]
Luc Barthelet, who I worked with at EA, is a Mathematica whiz (he later worked at Wolframe Research on Wolfram|Alpha), and he would prototype game ideas in Mathematica, which would render out web pages with animations.
He came up with a fun idea for a racing game renderer: it distorted the perspective transformation a bit, grading depth on a curve, so far away things would linger in the distance a bit longer, then speed up and WHOOSH past you, seeming even faster than they would be photorealisticly!
This may as well have been written in the stone ages, when we were banging AI rocks together.
I just did a ~6 month project in ~2 weeks using a frontier model.
I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!
ImprobableTruth 48 minutes ago [-]
>I just did a ~6 month project in ~2 weeks using a frontier model.
Claims like this are hard for me to take seriously because 'good' models have been available since the start of the year. So, if they really 10x one's productivity, then people should be able to have gotten done 5 years worth of work since then, but I've never actually seen anybody show any project like this.
34 minutes ago [-]
Shitty-kitty 26 minutes ago [-]
That's amazing. What's even more incredible is that somehow you managed to do a real code review and testing in that time-frame.
koe123 59 minutes ago [-]
Personal or work? Used by anyone or for fun?
37 minutes ago [-]
loveparade 52 minutes ago [-]
Used by Anthropic to sell tokens
suddenlybananas 1 hours ago [-]
What was the project? Could you share the code?
tombot 1 hours ago [-]
“16 developers across 246 tasks”
bfjvibybd6cuvu6 59 minutes ago [-]
With barely any experience or training with AI.
arisAlexis 39 minutes ago [-]
So why are all the top labs using the tools internally? They are lying or they are stupid?
Devs wish that was true but it isn't and it will get better.
dude250711 2 minutes ago [-]
AI psychosis is neither a lie nor a stupidity.
suddenlybananas 30 minutes ago [-]
Conversely, why do they have so many open bugs?
56 minutes ago [-]
bitwize 1 hours ago [-]
This study was shown to be flawed at the time; METR has retracted it. And it doesn't take into account current frontier models.
AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.
idle_zealot 56 minutes ago [-]
What tools? The ones that will be outmoded in 6 months, or the ones that will be banned in 6 months?
bitwize 25 minutes ago [-]
Whatever tools come along, step one is getting into the mindset that typing code in is no longer part of your job. You are a designer and director, not a coder. As Steve Yegge says, if you still have an IDE open entering code by hand, you're one of the crappy engineers. You need to be getting into the habit of understanding the strengths and weaknesses of your model and agentic harness and using those to produce the results you want. When those change in six months, you adapt along with them. Adjusting to the new mindset is the biggest hurdle, and there'll be plenty of devs who can't, and won't make the cut or stay in the field for very long. Just like there were plenty of devs who couldn't adjust to anything beyond COBOL on punched cards.
fxwin 52 minutes ago [-]
Have they retracted it? My understanding was simply that they released results with more recent data, not that this study itself was flawed (and their website doesn't mention a retraction either)
wisty 46 minutes ago [-]
Partial agree ...
I suspect:
If you know what you are doing it is a power tool.
If you don't know what you are doing it's also a power tool - if you measure a lot of devs then the bad ones (or anyone having a bad day, or the wrong fit for a project) can make work for everyone else at an outrageous pace.
croes 50 minutes ago [-]
So where are the profit jumps from all the productivity gains?
Or do you just produce more code but not more productive value?
METR already redid the study at a later date and now finds a likely 18% speedup
"For the subset of the original developers who participated in the later study, we now estimate a speedup of -18% with a confidence interval between -38% and +9%" (note their use of - and + here could be slightly confusing but they do mean 18% faster per the post)
https://metr.org/blog/2026-02-24-uplift-update/
I would not, at all, suggest that this second study corrects or debunks the first.
Instead what it shows (if anything, i.e. if you can even put aside the regrettable choice to change the payment level, which affects applicant recruitment) is that the mindset shift has already happened: developers now don’t want to attempt some tasks without AI.
What that tells you is not with any confidence that they are faster, but that we are possibly beyond the point that this can be meaningfully measured. AI could still be making developers slower, but developers aren’t going to be willing or perhaps able to help you find out.
Basically the job is different now.
What this does for me, perhaps, is vindicate my feelings. I can do agentic coding; I have learned the principles and some tools and I could learn more. But if this study is really reflective of how other developers feel now, I am done.
I do think AI has been a huge boon to productivity in many ways, but looking at feature timelines, I think it's pretty clear the 'critical shortest path' of key features hasn't been sped up by that much.
I think there is a simple reason for that. If you automate something, you make the measureable/predictable thing faster. So the hard to measure/predict part of the job will take more share of the time, and overall difficulty to measure/predict goes up.
I think this is what happened with Agile Scrum - as developers became more productive (for unrelated reasons, two main sources of SW developer productivity before AI were compilers and open source), the bureacracy (amount of meetings) increased, because the ratio of hard to measure vs easy to measure went up. Bureacracy is hard to measure, so it went up (as a share of work). I expect this only getting worse with more automation, such as AI. So I predict an increase in share of bureacracy compared to pre-AI world.
Either way, IMHO main point is automation has the opposite effect on human job predictability, it lowers it. Tasks we can easily automate are those that are easy to predict.
Overall this suggests to them that the current speedup is likely greater than what the study could measure.
Like, what people are saying is, “That old study was wrong! They did a new broken study that overturned it!”
Which is ancient at this point, and half a year older than the November 2025 inflection point when agentic coding got really good.
The original article is from August 2025, and the overall message to not trust ‘how it feels’ and rather measure outcomes seems right to me despite the outdated figures. On my team at least, we are seeing a noticeable inflection in work shipped with AI according to Weave.
A frontend dev doing tailwind integration for his day job is gonna see very different speedups than someone working in a niche scientific codebase. Taking the average makes about as much sense as taking the average of the speedup from calculators for a mathematician, a farmer, and an elementary school student.
That is, unless you're building a single page app/landing page that is the typical center column with a hero and below that a 3x3 feature grid with those same 3 colors that all the sloppers show off.
I'm not a frontend dev, but these statements are starting to get outright disrespectful to those that are. Do you people understand how much "world", customer and product knowledge is required to design and implement great UX/UI?
I promise you are not going to be able to translate all this internalized understanding to an LLM and have it do your "tailwind integration" It actually sucks at all frontend outside of the 3 types of page layouts it understand.. Shitty landing pages, generic dashboards and shitty blog layouts.
Ya'll yearn for slop though so maybe everything will just become shit anyways.
It's like saying you're convinced people reporting they feel more productive in a mauve-coloured room are liars, or those that drive automatic vs manual. Maybe they just find muave a restful colour?
Before, a backend guy asked to add an intranet page would make an austere page -bare html with barely any styling or javascript. Today, the same guy given the same task can turn in something with styling, javascript, internationalisation, interactive form validation, progress spinner, minification build stage, linting, maybe even automated browser tests.
And I have to code review it. Now the bottleneck of writing the code has been removed, I now find code review is the bottleneck - and a bottleneck facing much higher flow must either let more through, or start applying back pressure.
Sometimes I think an evil genie granted my wish for better tested code by trying to drown me in it.
Oh, the irony of this post being AI-generated.
For me as a dev, that's not the whole truth. Where I've found actual value in AI (and I think were some of that "perceived speedup" is coming from) is looking up things.
Unless you know the codebase and used libraries extremely well, you will have to do lots of "micro-lookups" during coding, where you have to find the specific APIs or library functions for your problem, then figure out how exactly you have to call them, how to handle the result, etc. That's lots of "research" work interleaved with actually writing the code.
AIs seem to be good enough to have a lot of that knowledge already baked into their weights, at least for popular platforms, so if you prompt it something, you can skip all that low-level lookup work or at least defer it until code review. Even during review, it's easier, because you don't have to come up with the appropriate library function from scratch, you only have to verify that the ones the AI used make sense and are used correctly.
That proves AI is capable of doing one part of the software engineering process. The 16 devs in the study trusted AI to write the code. Once we trust AI to do the verification as well we'll realise the gains we feel we're getting now. Essentially we're intentionally going slower on the second half because the trust is missing.
Alternatively, rather than trusting AI to do the validation, we could follow the vibe-coder approach by skipping the validation entirely, and trust that the generation stage is good enough not to need it. Historically that's come with some small downsides, like the code being a broken mess of security holes, but with time AI might fix that.
If you trust your team to care about quality then PRs aren't necessary, and if you don't then why are you trusting them to catching problems in PR reviews?
The reality is making good decisions and thinking about approaches take time. AI can absolutely make us faster at it but it's not magic and these speedups come with effort.
I'm British. I've been taught to turn understatement into an art.
[1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...
And this is coming from an AI sceptic.
Whenever I tell them about how awesome AI is, they come back with stories about how they used AI and it couldn't even do anything basic and what it did do had errors.
People will always create a world narrative that matches what they already believe.
Anti AI people are always quoting these "facts" about how AI reduces productivity even when developers feel it increases productivity - it reinforces their world view.
Productivity is not a feeling though. Either you show an increased productivity or it doesn't exist
The actual study with the data, minus the "I was right all along" commentary
ROFL sorry
He came up with a fun idea for a racing game renderer: it distorted the perspective transformation a bit, grading depth on a curve, so far away things would linger in the distance a bit longer, then speed up and WHOOSH past you, seeming even faster than they would be photorealisticly!
https://www.mobygames.com/person/29352/luc-barthelet/
https://community.wolfram.com/web/luc
This may as well have been written in the stone ages, when we were banging AI rocks together.
I just did a ~6 month project in ~2 weeks using a frontier model.
I wouldn't even have attempted this kind work a year ago, with or without the AIs available at the time!
Claims like this are hard for me to take seriously because 'good' models have been available since the start of the year. So, if they really 10x one's productivity, then people should be able to have gotten done 5 years worth of work since then, but I've never actually seen anybody show any project like this.
Devs wish that was true but it isn't and it will get better.
AI makes you more productive. This is no longer up for debate. The energy you spend arguing last year's talking points is better spent knuckling down and learning the tools.
I suspect:
If you know what you are doing it is a power tool.
If you don't know what you are doing it's also a power tool - if you measure a lot of devs then the bad ones (or anyone having a bad day, or the wrong fit for a project) can make work for everyone else at an outrageous pace.
Or do you just produce more code but not more productive value?