undefined

points

[-]

Unsurprising to those of us that has been screaming from the rooftops that human code reviews were barely catching anything in your typical corporate environment. The majority of the value was pointing out missing tests, because breaking bugs got past human eyeballs on a regular basis, even in very good companies.

It's not that the LLM code is that great, but that many overestimated the human code and the human review.

by userbinator5 hours ago|

prev|

[-]

I think it's more likely to be the same as average human-written code; which is what they were trained on, after all.

by vikramkr5 hours ago|

parent|

[-]

No it's far beyond average human code, and it's trained in RL environments on synthetic data etc now. The average human can't solve a millenium problem in mathematics but the LLMs are able to now. Anything with a verifiable output can be RLed on to make llms superhuman it seems, and the average human is not as persistent and thorough as e.g. gpt 6.1 sol at checking every edge case and hunting for bugs because that behavior is something that's just trained into the model. Frontier models perform farm above the average human at the categories of tasks they are RLed on like code and math.

by LtWorf3 hours ago|

parent|

[-]

> The average human can't solve a millenium problem in mathematics but the LLMs are able to now.

I think comparing the resources they spent at it would be a more apt comparison.

AKA if you tasked 150 mathematics departments to only focus about that problem, and stop doing things like teaching and researching anything else, you might get close to the amount of resources that these companies put into solving the problem to get a press release.

by RitchieRicher5 hours ago|

parent|

prev|

[-]

yet LLMs dont do off-by-one errors, typos mixing up variable names, interesting they seem unable to learn these human patterns

by FartyMcFarter1 hours ago|

parent|

[-]

They definitely write off by one errors. One of the first serious work tasks I used AI for a few months ago immediately had an off by one error.

What they are really good at is writing boilerplate and reading API documentation / example code. Avoiding subtle errors is not what they're good at in my experience.

by jacquesm4 hours ago|

parent|

prev|

[-]

I've seen both of those (and others besides). I don't know how much code you've looked at but your negative directly contradicts my experience. Review that code. All of it. Be super critical because it looks good but that doesn't mean it is good. Those are two entirely different things.

Also: most tests are positive tests. Tests rarely observe the negative space: that which is there but shouldn't be there.

by bigstrat20033 hours ago|

parent|

prev|

[-]

They do those things all the time. More often than humans, in my experience.

by thisismyanon24 hours ago|

parent|

prev|

[-]

Citation needed.

looks at particledb