It's relatively recent, I feel. Claude used to really struggle with writing asm not that long ago. But the last 6 months or so it's done great.

It's also far better than me (as someone who has done assembler since the Commodore 64) at using gdb to debug it, despite being effectively stuck using it in batch mode (which I didn't even knew existed). Watching it write elaborate scripts to dig into a code generation bug in my compiler is something.

I feel like the problem used to be that it'd struggle with the ambiguity of flow that is much more apparent in a high level language. But clearly that's not a problem any more.

I've had the opposite experience. Inspired by a recent HN post where someone made the world's dumbest browser by having an LLM read HTML and draw the rendered page, I thought I'd do a fun art project and take it a notch dumber: have the LLM "be" the computer and execute code as a VM. I chose a simple instruction set that I thought would be very present in the training material (RV32IM), and defined a very simple machine model with flat memory and two trapped calls (to terminate and to request to print).

I couldn't once get any of the SoTA models from a month or two ago to correctly execute more than the first 5% of the instructions for a fizzbuzz (compiled from C with GCC). As I recall, one of the Qwens did the best and would only mess up "a little bit", but that's of course enough to derail everything (can someone remind me again why we think natural language is good for interacting with precise machines?). I didn't think it'd go very well, but failing at decoding something as well-documented as RISCV is not very impressive!

Most models would also start gaslighting me when I pointed out their mistakes. To their credit, they'd very often cheat by deducing that the code was for fizzbuzz, and try to fake the execution. Always badly though. (This despite explicit instructions to execute the code faithfully instruction by instruction and not be informed by their overview of the code).

I honestly don't understand how people can work like that. I had fun because the whole thing was a joke, an art project. Doing serious work in that way must be so ridiculous.

But then again, I don't use LLMs very much and might be holding them wrong.