upvote

points

by 011000117 hours ago |
comments
upvoteby throwaw127 hours ago|
[-]
> we'll be quite busy in the meantime fixing all the bugs that AI finds.

Maybe that's the job of software engineering moving forward.

Client: Hey, we have got these 125 microservices created by our agents and for the last 25 days they got stuck and can't add any new feature without breaking things, can you take this?

Eng: Sure, lets sign a 24 month contract, my rate is 250$/hr

Client: Sounds good

reply
upvoteby impossiblefork24 minutes ago|
parent|
next
[-]
Eng: So I prompted Claude to turn the 125 microservices into a single Erlang program. Better not tell the client.
reply
upvoteby badRNG6 hours ago|
parent|
prev|
[-]
Do you think that even today's frontier models are unable to handle this task?
reply
upvoteby Aurornis6 hours ago|
parent|
next
[-]
Today's frontier models can't even handle 100% of the SWE benchmarks that have been around for longer than they've been training the models. Companies that are benchmaxxing their models against the benchmarks haven't even been able to get them to 100%.

I've been using Astra and Opus 5.5 all week and I still have to intervene and tell them to do something different all the time.

I think the people whose SWE jobs consisted of simple features, bug fixes, or tweaking .toml until the server works are on their way out.

Someone needs to steer the LLMs around for now, though. There is a very long tail of non-trivial work and expertise that can't yet be replaced by a CEO telling the LLM to make the product work. There are many CEOs trying to do that right now and, outside of very simple CRUD apps, it doesn't work yet.

reply
upvoteby PunchyHamster1 hours ago|
parent|
[-]
> Today's frontier models can't even handle 100% of the SWE benchmarks that have been around for longer than they've been training the models. Companies that are benchmaxxing their models against the benchmarks haven't even been able to get them to 100%.

Is average developer 100% the benchmarks ?

reply
upvoteby criemen5 minutes ago|
parent|
[-]
Given how flawed the benchmarks are (see for example https://epoch.ai/benchmarks/deepswe/review), not even the mythical 10x developer is gonna get 100% on these benchmarks.
reply
upvoteby skeptic_ai4 hours ago|
parent|
prev|
next
[-]
Well, I let him do some stuff in kube. The architecture is a complete mess.
reply
upvoteby sroussey6 hours ago|
parent|
prev|
[-]
Today’s frontier models create the problem.
reply
upvoteby falcor846 hours ago|
parent|
next
[-]
That isn't actually relevant. They create problems, but when confronted with them and asked to fix them, my experience is that they can fix them very effectively. I haven't had an AI session get "stuck" on me, not being able to solve a problem in over a year.

The challenge as I see it is in shifting checks further left, but actual mitigation is just not really an issue any more.

reply
upvoteby WithinReason6 hours ago|
parent|
prev|
next
[-]
What problem? They keep finding security bugs that humans couldn't for decades.
reply
upvoteby knowaveragejoe6 hours ago|
parent|
prev|
[-]
So ask them to fix the problems in a targeted way.
reply