It can equally hallucinate commands. Fine. The problem is, if I'm working on a remote machine, I'm generally doing things that I'd be less concerned about. If I'm on a VPN and it rm -rf / while I'm trying to clean it up; bad break, but it's not _my machine_ it just removed root on.
So if your LLM is just running something like `ssh <remote> "<cmd>"` it could easily foget the ssh <remote> part and suddenly you're modifying your local system.
So it's one thing to YOLO on production servers, etc, but wiping out something locally is a significantly different event. Imagine it erasing all your scripts or whatever.
Anyway, the point is: I wouldn't trust an agent operating with just a bash cli running ssh commands.