I suspect (with zero proof or understanding) that this has something to do with how well C maps to assembly. It's not a stretch to say the model's vector space maps this chunk of assembly with that line of C. And we all know how much C code exists online.
It's also far better than me (as someone who has done assembler since the Commodore 64) at using gdb to debug it, despite being effectively stuck using it in batch mode (which I didn't even knew existed). Watching it write elaborate scripts to dig into a code generation bug in my compiler is something.
I feel like the problem used to be that it'd struggle with the ambiguity of flow that is much more apparent in a high level language. But clearly that's not a problem any more.
I couldn't once get any of the SoTA models from a month or two ago to correctly execute more than the first 5% of the instructions for a fizzbuzz (compiled from C with GCC). As I recall, one of the Qwens did the best and would only mess up "a little bit", but that's of course enough to derail everything (can someone remind me again why we think natural language is good for interacting with precise machines?). I didn't think it'd go very well, but failing at decoding something as well-documented as RISCV is not very impressive!
Most models would also start gaslighting me when I pointed out their mistakes. To their credit, they'd very often cheat by deducing that the code was for fizzbuzz, and try to fake the execution. Always badly though. (This despite explicit instructions to execute the code faithfully instruction by instruction and not be informed by their overview of the code).
I honestly don't understand how people can work like that. I had fun because the whole thing was a joke, an art project. Doing serious work in that way must be so ridiculous.
But then again, I don't use LLMs very much and might be holding them wrong.
(Oh, and the "compiler" will also refuse to generate certain types of programs.)
How can a generic LLM generate better assembly than a dedicated compiler, whose sole purpose is to generate assembly code. With people pedantically adding every optimization imaginable and unimaginable to produce the most efficient code possible. And you have the audacity to say LLMs, which write garbage non-trivial amount of time, are capable of producing better assembly.
This has got to be either a masterful ragebait, or a person with very low knowledge of modern compilers, because even an LLM would not write something so stupid as this.
LLMs generating "assembly that runs a great deal more efficiently" is a ludicrous claim that cannot be substantiated outside PEBCAK situations.
I've had all my side projects being written in x64 for the last 6 months and it is shockingly effective.
Have you ever compiled something by hand? You should try sometime, it's an illuminating experience. Humans find it hard because you have to remember a lot of details while simultaneously paying attention to a different large set of information while also generating instructions. It's tough, but not impossible, it takes humans a lot of time and effort. How might a computer fare if it could remember everything and pay attention to multiple inputs and outputs at once? That's what an LLM does.