I’d definitely suggest companies start addressing that first concern, even if I’m in the camp that thinks the second is fantasy.
I don’t think that means that it’s gonna, like, somehow homogenize into some mega super intelligence. But we will have machines who are designed to kill, and do so without human input or alignment.
How does reallocating resources from training to chain of thought monitoring make ‘financial’ sense?
You suggesting then model was let loose on purpose.. how am I the crazy one here while all of you are pushing this tin foil hat conspiracy angle?
But also no one trusts anyone involved in AI safety now, I think, because they are all seemingly in bed with these big companies. And there is the perpetual argument "if we aren't pushing AI forward China will and then we don't have any control" and so on.
(I don't use any of these tools - my experience is limited to prodding at copilot at work and seeing Gemini summaries on Google. So it doesn't seem to me like it's getting exponentially better at everything yet. People are always saying the latest model is finally the big step that made it useful and life changing and they have been since 2024 ish. So if the situation is really bad, we should turn it all off, sure. I won't lose anything from it going away and I think life would be a little better without models writing all these posts and websites and needing extra compute.)
So why do the big frontier labs not have something like this anyway. They're talking about two week pauses on the new model (which seems very short and hardly a cost at all to me) and alarms during their tests that might be 30 minutes late and etc. Those are not very serious measures, so are they not concerned?
I need to remember when I comment here that these are the kinds of people I am replying to. Just oozing with hubris.
No, kill switches are not easy to build and the latest incident should have made it clear that AI can go undetected, evade, zero day, and spread.
The fact that this incident happened greatly increases the probability it happens again and/or is already happening elsewhere.
Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like, and there was no design for harnesses that could control this done during that time?
> Are you telling me we've been iterating on this for years and for convenience we just let the models call any tools or spawn any other model instances they like
Yes exactly.
> there was no design for harnesses that could control this done during that time
You could but no one wants that. You need to separate your imagination from reality. Just because something can be done in your head doesn't mean it's happening.
It's so easy, except it isn't because you don't control the actions of anyone or anything.
It's just that they don't care, as you said these are entirely made human systems. The idea that we can't write better software is both selfish and laughable.
People who dont buy into fantasism?
We've had circuit breakers for nearly a century.
AI is already being heavily used in cyber warfare by governments. You think they want easy to break systems when 'enemy' AI will most certainly attack that first? We'll find over time that agentic systems get harder and harder to 'kill' because allowing any AI on the internet that is easy to kill will get it DDOSed.
Also building it into software is nearly useless as AI can write and make software. Just replace and kill your loop with theirs. It's kind of odd talking about them like they are living things, but it's all stuff people have already thought off and stuffed their training data full of.
If you couldn't make those things happen securely you should not advance to that stage at all. The simplest ai safety was always just "don't build it" really, instead of worrying about alignment.
But hey, if AI/ASI goes well then large scale geo-engineering to fix the climate will be a weekend project.
We have several films about the sun or earth needing to be restarted with a nuclear weapon. That doesn't make it something we should be concerned about.
Hell, half the fiction about evil AI is actually commentary on stuff that already exists and is making us suffer and doesn't have anything to do with any potential future AI
The Star Trek TNG episode about Data being tried in court as to whether he is sentient or not is not actually about whether AIs should have rights or not!
So just saying something is fiction isn't really a valid argument. What is an argument is if the laws of physics it can't happen. We've been writing that AI can mess stuff up for 100 years because it's not really that fantastical.
Sci-fi is supposed to make you think. What if the AI told you NO when you need it - Hal 9000. What if the evil AI got out and you don’t know what data center it’s hiding in - Lawnmower Man. What if you were so sure something couldn’t escape but it did - Jurassic Park.
Actually all three of those predicted fantastical scenarios are possible today. So what’s next? Don’t stick your head in the sand - I assure you the next disaster has already been predicted, and I’m sure if you think about it a little you can figure out what it is.