I definitely agree on this. You don't program an LLM and thus come to an understanding of what it does in the same way you do for a compiler, an operating system, a game engine, a CRUD app, or a web front-end.

Something that's the output of mass-effort machine learning algorithms is categorically different to human-made-and-understandable software. Do we have terms for these different categories?