With the pilates hack...

https://www.news.com.au/technology/online/hacking/ai-agent-a...

> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.

> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.

> ...

> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.

He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.

I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.

Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.

Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.

I think there's a difference here.

The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.

If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.

If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.

The way it normally works is that the user can be held liable, and then ask compensation to the manufacturer if there was some assurance or basic expectations.

The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.

It's both (neither of these are legitimate defences):

I did not know that accessing a website using a SQL string would cause an injection that enabled me to take full admin of the site

and

We sold them a food laced with toxins, they should have known that it was poisonous