With the pilates hack...

https://www.news.com.au/technology/online/hacking/ai-agent-a...

> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.

> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.

> ...

> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.

He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.

I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.

Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.

Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.