This goes both ways. You absolutely can constrain an AI system by “putting it in a box”. The point parent comment was making is that, such a device is borderline useless for its creators. Why invest trillions in capital on a system that can’t even accept input from the internet. So you set it up with an ethernet connection. And this is good, but you have a hardware failure at the concrete room data center. That’s pretty annoying for your customers, so you install some doors (with electronic key codes of course) and give a bunch of (trusted, vetted) people access to deal with those. And this is fine, but it turns out some of your customers are having latency issues so you build more data centers with more humans granted access to copies of the intelligent system. And this makes people happy but to get a faster feedback loop your customers ask to let the AI system have more permissions to the system they’re operating on. And they come with billion dollar checks, and the system hasn’t harmed anyone yet, so you say, “Okay.” And now you find yourself where we are today where AI systems can remotely run arbitrary commands on thousands if not millions of systems, where many individual humans with all of their frailties and idiosyncrasies can physically interact with the hardware running these systems, and where there’s an economic demand to tighten the loop between action in the real world and a response by an AI system. It’s very obvious that the story doesn’t end here, so where does it stop?

If you want to argue about current existing models (you mention Claude and problems modifying images), then sure, I'd agree with you!

The issue is not current models, but straightforward engineering evolution of them. It's like looking at the Wright Brothers plane and saying "sheesh, that will never get me from New York to Paris in 4 hours, that's just fantasy!" And remember, airplanes do not accelerate their own engineering, whereas pretty much all AI labs are already benefitting from AI in their own work to develop AI.

If you want to argue that no matter how much you engineer it, it will never be as smart as a human let alone smarter, then make a specific argument for why is that. I think you'd still be wrong but at least it would be interesting: ) But saying that you can defeat an actually smarter-than-human AI by just pulling the plug, because current models can't get a picture always right, is not a valid argument.

Drugs need no will or intelligence at all to cause addiction, and similarly, AI does not need to be an evil genius to become overused and destructive.

It also doesn't have to be just one (humans hurting themselves with passive AI) or the other (selfish AI hurting humans). They'd work great together.