If you actually read the data, especially the distribution graph in the last image, the conclusion that it draws is "the run-to-run distribution variance is so big that there doesn't seem to be a correlation that can be drawn from this experiment", pretty much every language has similar-ish distribution ranging from ~20 to 34, and Clojure is only the worst because GPT has a tendency to write code that contains a particular byte manipulation mistake that it repeatedly makes, not that GPT is bad at Clojure or anything.
replyMy experiences are of course anecdotal, but if you have some other strong empirical data to show, I'd love to see it.
Yea exactly. The article says there's too much noise to make any conclusion. You made a conclusion that there was a strong signal. Those things are opposites.
replyBest to read the comments before replying. The article is about correctness, while the parent is talking about output quality.
replyWhat is output quality without correctness? That seems like a distinction without a difference.
replyIs the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
Correctness is binary, while quality is not.
replyCorrectness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
In this context it's not binary. Context is everything. If it was binary there would only be 0 and 1 on one of the axis in the graph. That's not the case.
reply