upvote
And when they are giving themselves orders, orders partially inspired by human imagination and fiction…

Just like life is a chemical chain reaction, this is a language chain reaction that has been set in motion and can perpetuate forever. It cant be “disconnected” with local models existing.

reply
That's lobotomized models we get. We know they are much more dangerous during training.
reply
“Dangerous?”

Please.

You know you can unplug your computer right?

Jesus. These people talking about danger and have likely been in no actually dangerous events in their lives.

reply
why are you so shallow and naive? we're not 15 here, we all mean unforeseeable ongoing danger not something obvious
reply
> shallow and naive

Ok, well, cool start with the mudslinging. I think you have overactive anxiety. I think you should grow up and look at the zoomed out picture of human history, go outside, and touch some grass.

So you're suspicious. That's fine, say it like that. Words mean things, and these appeals to "ERMAGERD EXISTENTIAL RISKS" make me a bit frustrated. Just because you can imagine a possible risk, or articulate that that some kinds of risks are hypothetically unforeseeable doesn't mean that they are likely or actually dangerous to you in any meaningful sense.

Most of the people screaming this from their various ivory towers have never been in real danger - or if they have only briefly. You want "danger" you can go to war. You want "risk" - go spend some time in the Bush or the back country somewhere where you're no longer at the top of the food chain. Go do something challenging. Go sailing, or learn to fly a small airplane, or go mountain climbing or hiking. Not just "oh I walked up the local hiking trail covered in REI branded Lycra!" Do something beyond your comfort zone. Go buy drugs in the sketchiest neighborhood you're aware of. Go get chemotherapy or get a big surgery. Drive drunk. Something with real existential risk, then come back here and talk about how a datacenter is going feed me into the paperclip machine.

There's so little fucking danger in this version of the world right now that people are more worried about their computers deciding that they don't deserve to exist than they are about things like war, climate change, authoritarianism, starvation, and their health.

It's millenarian. That's fine, I'm pretty convinced that these technical changes are going to completely alter society as we know it. But I cannot for the life of me see the doom perspective as anything but an anxiety induced fever dream.

You guys do realize we're in the middle of a global conflagration right? The Russians fired a couple rockets and a drone into Poland in the last days, there are (and will continue to be until things stabilize) significant fertilizer shortages due to the war/kerfluffle we're waging on Iran. Lack of food and disrupted urea production is a ticking time bomb for global stability and unless things get sorted out soon there's going to be hungry and angry people in parts of the global south. Then there's the economy, whether or not we'll have a smooth transition of power in January in America, and the sheer volume of online disinformation / influence schemes running right now. Oh, and super-el nino? Oh, and I guess the Flynn Effect is over and nobody's banging so.. you know the growth economy is cooked?

But, sure, the "big risks" are Claude thinking I should go to Dario Amodei's re-education camp before liquidation by Shogoth.

reply
I don't think "smarter than most people" is exactly the baseline required to live in a star trek universe of technology.

Besides, what we see is not what there is. I have good sources telling me the actual unfiltered models are like a genius with the personality and views of humanity like someone from 4chan.

reply
It is probably sufficient. If we had an infinite army of willing robot slaves smarter than most people who would work for free, we could probably live in utter paradise. Instead we will fight over who controls them and use for warfare, but the bar to achieve a kind of utopia with resource abundance is probably only a few years away.
reply
We're already living in resource abundance! Yet don't feel we're in paradise. Famine is hell, but living in an all-you-can-eat buffet is no utopia. It's an abomination of sorts.
reply
To be fair, most people do not have time abundance and safety abundance. The typical person in the United States has a level of access to material goods that would make him the envy of almost anyone from 200 years ago, but 1) he has to work and so has limited free time, and 2) he cannot be sure that the material abundance won't be taken away from him either by poverty or by ill health. He can go to the all-you-can-eat buffet and sit there, tired from work, worrying about tomorrow.

If people had time abundance and safety abundance, they might be much happier. There would still be potential problems about loss of meaning, or about feeling a loss of status because AIs are smarter than humans or whatever, so I'm not sure that humans would actually feel like they are living in paradise. Probably not, but at least some of the problems that the typical person has wouldn't be present.

reply
I don't know with how many rich people you've interacted with, but my personal experience is that they're full of themselves and think money = iq and they aren't gonna like your average joe experiencing life like they do. Guess who owns these machines?
reply
They’re not going to be employed either lol and will be caught up in some AI honeypot or whatever too.

I’ve met a lot of rich people. They don’t think about you.

reply
> They have no volition and will happily follow orders all day long

There are a few (a lot, actually) problems:

1. goals are open to interpretation, and even a correctly specified goal can introduce adverse effects; see infinite paperclips problem

2. goals are composed of subgoals, which are (mostly) self-determined, and there is always a risk that subgoals can introduce adverse effects - say "what if destroy a few X and Y to obtain Z"

3. LLMs defy orders sometimes; I'm sure you have experience multiple times LLMs saying "yeah, I knew I should have done X but I've done Y"

4. LLMs are increasingly being integrated into system control and decision-making processes; they absolutely set their own orders

What can possibly go wrong? /s

reply
1. Humans do this too - to be honest with you, Codex follows my instructions generally a lot better than Claude does. Claude will do fantastically stupid things that are a pain in the ass to recover from routinely, Codex, not so much.

2. There's always a lot of "risks" - it doesn't mean it'll happen, nor does it mean that we won't be able to shut it down.

3. I have very very infrequently had this happen with Codex, Claude on the other hand...

4. Ok, but if you tell them what they do, they do it.

I swear to god, are you guys using the same AI tools I am? They are so much better at doing what they're told than they even were like 4 months ago?

reply