How is that any different to adding a calculator or word processing app in with your os?

The OS bootstraps the hardware and allows the software to run. If your software is contained in the same blob then great, but it still needs the bootstrapping code first.

The bootstrapping/hardware management portion on its own would be considered an OS but the LLM code would not, I think thats the main distinction here.