For actual output quality, the main deciding factor is simply how much tooling it is there for the LLMs to check their own work, as LLMs seemed to avoid using a lot of libraries in general. That's why C# is underrated due to the tooling strength of the .NET ecosystem, as long as you tell LLMs to avoid using reflections unless absolutely necessary.
C++ is also surprisingly good, but you pretty much have to tell the LLMs to treat it like Go and don't use any of the dangerous features for normal code.
The need to select an appropriate 3rd party library represents an entire dimension of the search space that can be eliminated. Imagine having to make this choice multiple times per day when your competition is just mindlessly using System.* types. The fact that the .NET ecosystem is curated by one entity should not be underestimated.
Even when we do need to import 3rd party nugets, the models seem to follow this highly structured pattern. They scan the xml docs, and failing that they will build a throwaway console app to reflect over all the unique types and build a report. The fact that we can easily do this with a simple powershell command makes a big difference. How many other ecosystems can even consider doing this? Reflection is a superpower, not something to be avoided.
Reflection is just such a dangerous feature that looks like ordinary code, which is why it is something to be avoided, and having an LLM write/analyze the code for eliminates the need to use a lot of reflective code to begin with.
Another slept-on feature is the ability to quickly write your own static analyzers, which along with the myriad of existing analyzers out there, provide excellent guard-rails to prevent slop.
For existing codebases I made the experience, that LLMs are very good at replicating their style.
At work most of our C++ codebases use a fairly consistent style and subset of C++ features and to my initial surprise specifying style conventions etc explicitly turned out to be mostly superfluous.
Of course, we also have some legacy projects originally, written in ANSI C, which only received a few changes in the last 15 years to compile with a C++ compiler. Here a style guide is helpful, bit I consider it more like a temporary instruction for refactoring.
However, I struggle as well and the error messages of Oracle are some of the worst I've ever experienced.
And yet this article has pretty strong empirical data to show that your intuition here is incorrect. You should back up your statement with something more than vibes.
My experiences are of course anecdotal, but if you have some other strong empirical data to show, I'd love to see it.
Is the claim that LLMs produce Go code that is superficially nice looking but in fact fail to solve the stated problem? Because that's an anti-Go position I'd say.
Correctness is a suitable property to act as a multiplier in your formula, where incorrect is 0 and correct is 1, but you also need other facets to find a quality gradient.
I’ve had good enough experiences with Rust, but actually OCaml has been hands down the language I’ve seen best results with. The quality (and performance) of code is just phenomenal — and the main issue when working as a solo human with the language, namely smaller pool of community libraries, just isn’t an issue any more. Jane Street has really done tremendous work modernizing the language and tooling.
> This is very different from Python, where training data is polluted (I presume) with tons of code written by non-software engineers and demonstrating many different ways of doing the same thing.
Counter-example: agents with Django-related stuff. Excellent output.
LLMs seem to fall apart where human written projects of the same nature had no particular way about them. It is especially apparent when treading into waters where beginners are found. Like the earlier comment suggests, this is presumably because the LLMs struggle to find any kind of pattern to latch onto. Django offers a pattern, but one not shared by rest of the Python ecosystem. Whereas virtually all Go codebases look the same.
> "They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt."
This makes it a great language not just for young Googlers programmers, but also for LLM Agents!
IMO, the next big language will be similar philosophy, but without garbage collection. (is Zig the closest to filling that niche?)
Given what you said about Go, presumably that is Solod (https://solod.dev)
Python's philosophy is there is one way to do it, as opposed to Perl's TIMTOWTDI.
Your statement also assumes that 'software engineers' write the best code, and from my experience, this is definitely not true
I believe the training data should simply be limited to only code written by someone like Fabrice Ballard, or whoever you think writes the best code.
> There should be one-- and preferably only one --obvious way to do it. [1]
Note the `should` and the `obvious`. Is it not a strict rule about having a single way to do things. It is about the aspiration that, if you do something, there is single obvious way to do it, much better than the others.
(I agree though that not even this is true anymore, see how many different ways you have to interpolate strings).
As an example, there are dicts, tuples, classes, NamedTuples, dataclasses, attr.s, Pydantic, and pretty much all of those solve similar problem (hold my data) but have slightly different properties and use cases.
Also, if you trust the benchmarks, it seems that Python is, at the very least, decent enough for LLMs. There seems to be "no trouble" in practice, unless you show us better proof than "I feel like it must be bad for this and that".