It's not deterministic due to implementation details in floating point numbers and order of execution.

If you ran an LLM with infinite precision and guaranteed order of execution, it would be deterministic.

(I think determinism is overrated. Being deterministic does not make LLMs more reliable or correct.)

These are all implementation details.

At the end of the day, an LLM is just a very big mathematical function. That is, by definition, deterministic. A particular implementation might give up on determinism for the sake of higher efficiency, but it you want a deterministic LLM, it can absolutely be done.

Yes, yes, assuming the same CPU and stuff. There is some randomness due to floating point math differences between CPUs (and sometimes on the same CPU), but I don't think that fundamentally changes my point.

Put another way: if you could have a virtualization layer that guarantees deterministic floating point operations then a LLM set to 0.0 temp would produce deterministic output.