I asked them to do a thing, but didn't intend the obvious consequences* so it's not my fault they occurred.
And now we have the same thing but the bosses 'hire' AI.
Now I realise this is part of how unusual my thinking is.
I'm happy to use phrases like "ChatGPT hacked out of the sandbox, then hacked into HuggingFace"; people often respond to this like I'm suggesting OpenAI isn't at fault, and like, that's not my position at all, so far as I'm concerned the buck still stops with the person who set the task regardless, the thing that changes from incidents like this is now nobody in the future gets to even have the excuse "oh but we didn't know it could even do that" or "we didn't know it might interpret our orders in that kind of way".
The response, both when a human messes up and now when an AI messes up, needs to be defence in depth: someone giving orders needs to be giving clear orders, entities (human or machine) who follow instructions need to have not just an understanding of how to follow them, but also what's so out of scope as to be forbidden - the difference between 'follow orders' and 'follow lawful orders'.
That’d be engineering.
Real world, as you say, not so simple. Everything has to deal with certain degree of forecastable nonsense, e.g. a bridge has to cope not only with traffic and winds, but the possibility that someone will be drunk in charge of a ship and crash into it.
> AI is built on human knowledge so guess what it will keep doing.
Yes, and also brings its own additional mess on top of that. All machine learning takes a huge number of examples to get good, so an LLM isn't just "read all the online courses in how to run a business", but also likely has 50 business versions of the recent demonstration of common sense failure with "I live 100m from a car wash, should I walk or drive?"
> How do we build systems without assuming complete adherence, but tolerating imperfection and failures? Isn't there some discipline teaching us that?
Many such disciplines. Perhaps all except maths and computer science? Or even including maths and computer science, given stats is part of maths and even compsci has to deal with fault tolerance.
Of all the things that failed for that leak, we should focus the LEAST on the password being insecure, and the company whom had their account misused, and the most at the other end of the long line of failures.
Why was there no monitoring on a company suddenly looking up 600 people a minute, why was this only discovered when they were making the invoice?? And how was it even possible to have a password that unsafe, no two factor auth etc etc etc.
As for 2FA, it is a nice thing to have, but it comes at a significant support cost. People lose their token, people get annoyed by the friction, people can't figure out setup (especially older folks).
Government services have to work in all these scenarios, simply because that is a right of the citizens.
Our system now heavily reenforces lack of accountability to executives for what happens under their watch.
Investors don't lose money when these breaches happen.
This is why it won't change until those change.
Otherwise shareholders do not care, because they do not have skin in the game.
Same for government staff. Unless they are explicitly fired there are no consequences of abusing the trust of public.
... said every "security" pedlar ever.
> I dont understand why there is not massive reorganisations in systems when things go wrong. Its always , oh yeah so we fired the guy furthest down the ladder.
Because massive reorganisations can easily lead to even more things going wrong. Also most people are lazy and phlegmatic by default.
I think there's also a big part of the line that Jennifer Lawrence says in the satiric comedy "Don't look up":
"They are not even smart enough to be as evil as you're giving them credit for."
Just as the person who picks 123456 as a password doesn't see where the problem is, I think there are really quite a lot of people dumb enough all along the decision chain to think that firing the person who reported the problem did actually fix the problem.
They are really that dumb.
This would just replace one insecure system with another.
It is time to recognise there's no such thing as a secure connected computer. And thanks to "AI" there's no such thing even as a significant defence lead over attackers.