If you're relying on asking the LLM "pwease don't delete" then you're already in trouble. This kind of stuff doesn't work with people either and they generally exhibit actual signs of intelligence.
replySure; it's an example, I would never rely "Please don't delete" for real important data. However, we're being sold an idea that we should just let the agent do everything, so I think it's important to point out how utterly insane that idea actually is.
replyAbsolutely. It should simply
not have permissions to delete your production database, can’t rely on prompting alone for that kind of safety.
replyThese are still stochastic machines, guardrails must be inserted at the system level.
They are getting better every day about managing their own guardrails, so we will get there eventually.
I entirely agree, although I think this is counter to the narrative that we should just let the agent do everything that the labs want to sell.
replyThere's a really major problem with the concept of human-in-the-loop though, which is just that humans are not built for that that kind of work. It's like all those tests against the TSA where they manage to sneak something through that should have been caught. But the problem is, a TSA agent sees probably like 1000 things they can ignore to the 1 thing they need to look at, and it's easy to just start rubber stamping things.