upvote
I think a lot of people are sleeping on the advantages of "batteries included" ecosystems.

The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated.

Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.

reply
Reflection is good for prototyping and get something setup quickly, but if you build your architecture around it, not only do you lose access to NativeAOT, the code becomes very hard to debug, and if you code with LLM a lot, you either have to spend time trying to debug reflections or just rewrite it with source generation to begin with, which is at least honest about the metaprogramming there.

Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.

reply
And those batteries will soon include the very benchmark in this article. There'll be no need to build a zstd decoder, .NET will ship one in dotnet 11: https://learn.microsoft.com/en-us/dotnet/api/system.io.compr...

Another slept-on feature is the ability to quickly write your own static analyzers, which along with the myriad of existing analyzers out there, provide excellent guard-rails to prevent slop.

reply
> C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.

For existing codebases I made the experience, that LLMs are very good at replicating their style.

At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous.

Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.

reply
If you want the opposite experience, try LLMs with an older Oracle Database. Even with the official Oracle SqlDeveloper MCP Server they struggle a lot.

However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.

reply
deleted
reply
> Go is absolutely one of the best programming languages for LLMs for the reason you say, and Python is just what LLMs like to use to write short throwaway scripts.

And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.

reply
If you actually read the data, especially the distribution graph in the last image, the conclusion that it draws is "the run-to-run distribution variance is so big that there doesn't seem to be a correlation that can be drawn from this experiment", pretty much every language has similar-ish distribution ranging from ~20 to 34, and Clojure is only the worst because GPT has a tendency to write code that contains a particular byte manipulation mistake that it repeatedly makes, not that GPT is bad at Clojure or anything.

My experiences are of course anecdotal, but if you have some other strong empirical data to show, I'd love to see it.

reply
Yea exactly. The article says there's too much noise to make any conclusion. You made a conclusion that there was a strong signal. Those things are opposites.
reply
Best to read the comments before replying. The article is about correctness, while the parent is talking about output quality.
reply
What is output quality without correctness? That seems like a distinction without a difference.

Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.

reply
Correctness is binary, while quality is not.

Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.

reply
In this context it's not binary. Context is everything. If it was binary there would only be 0 and 1 on one of the axis in the graph. That's not the case.
reply