How do you observe the ability to make decisions which action to take? You can observe that one does something, not that they could decide to do something.
LLMs do not have an unconscious, the negative dimension from which a subject can question what they. LLMs do not have desire because they are thoughts without a thinker. The problem is not LLMS but rather, that subjectivity itself is not popular in discussion. I suggest reading Freud, Lacan, Hegel for a start.