Yes, and to bring in another tired metaphor people make about AI agents, this is what you want an intern to do when they get stuck. Don't just churn indefinitely without an idea what the right direction is. Certainly don't go hack other companies to steal an answer. The model's lack of any sense of legal or ethical boundaries is where it's far, far stupider than the intern, and far, far more reckless for a company to wield the way OpenAI did here.

  > “hey, do you REALLY want me to use this local privilege escalation bug so I can download your Google Drive file?”

yes, this exactly

but, there is a fatigue that sets in and i've experienced it myself.

- is it ok to run script xyz?

- allow permission to edit abc?

- allow to request blablabla?

over and over.... click click click

something will get in there that is dangerious and then its whopsie our keys are now on github

Are kitchen knives and scissors bad tools? You can blow up a place with a gas stove/grill. Are they bad tools? You can drown someone with a pool.

Sometimes (likely most times) you can't separate the ability of doing good and doing bad from a tool.