By no means I'm saying this isn't impressive, but I think this is another case of LLMs being good at translating semantic meaning between different representations.

Obvious ones are spoken languages and code, but they can also natively interpret some simpler ciphers, hashes or simpler encodings, partly (I think) because it was in their training data.

As others have postulated, if they've been ingesting raw weights, naturally they learn how to translate semantic meaning to matrices.