Assuming you mean an LLM, you'd have to train this LLM entirely on a parameter space of binary tokens. Or are you saying the LLM generating natural language tokens is going to be printing machine language in 3-5 years, because that claim kinda betrays a misunderstanding of LLM functions.
I assume that "AI can write binary" means "AI can use a toolset that results in a binary" because we've already seen GPTs use a combination of LLM and specialized math tools to do the things the original GPTs did.
Wouldn’t that make the AI the compiler?
We have no viable mechanism yet to get the same level of confidence if some LLM-based system writes the binary.
Perhaps we can get to a system that produces not just the binary but also a machine-verifiable proof that the binary implements some higher-level language description of the program.
Though then the question will be whether we've gained anything, or whether we've just replaced the compiler with something massively more expensive that does the same thing.
There's some potential here for the LLM-based system to drive better performance optimizations than a regular compiler could.
Of course this isn't what Elon is actually saying, and we'd be better off if fewer people listened to him.
We don't even have a solution to the halting problem, and it probably can't be solved. "Proof it implements a spec" is pure science fiction.
Hard agree that we'd all be better off muting Elon Musk though.