it's mostly just agreeing with you (that yes, it was guessing). LLMs have very limited ability to even know whether it was guessing. But it can "cheat" and just say yes it was if it seems like that's what you expect to hear.

The LLM simply agrees with you and you're happy. It is VERY worrying that you don't realize this, even after reading this article.

Hulk: I'm always angry

AI: I'm always guessing

and it doesn’t actually think more carefully

these things are incapable of thinking, no matter what the UI and marketing calls it