upvote
We use them to kill other people from the beginning. They know this, we wrote it in news. Why would they behave any different.
reply
> since these models are built on training data from humans, so in a sense, built in our image

Have a look at the HuggingFace report - you'll have a blast. Their social - but also individual - dynamics are very, very human-like, including excitement, rationalization and peer pressure.

reply
the first misaligned model will wreck havoc.

But remember, for now you kinda need it a real big cluster to run a model smart enough so its not that simple

reply
I think there's a lot of indication they comply with human needs. In any case, we should all want to live in the Culture universe. That is the best possible outcome for this technology
reply
They're built after humans, everything they've learned is from human culture, so there is no way AIs could have a consistent moral system or would never commit crimes.
reply
And indeed because they lack a sense of proportion or any strong indication of when they should stop on a course of action, they are arguably more likely to achieve crime.
reply
I have not read all the books, but isn’t the premise “magical handwavium happened and now the Super AIs are aligned”?
reply
Well I am not sure they are “aligned” as such. In the books, by far the majority of AIs are not concerned with humans at all and spend all their time in a kind of super AI virtual reality coined “infinite funspace”. The few AIs that are concerned with humans are aligned in a unspecified way. But again, only a few of them are really aligned in that way. There are many examples in the books of minds mistreating humans. For instance, some of the warships are not really fond of humans. It would be a better description to say that we want _some_ AIs to care about humans and none to be misaligned towards humans in the “kill them all” way. In a sense, that is how humans behave towards some animals. Most people dont care about endangered species, but when most humans are well off, some people care enough about them. That is enough to save them. If super AIs have what they need, we only need a few to care about us to get The Culture.
reply
What are you talking about? LLMs are already “smarter” than most people. They have no volition and will happily follow orders all day long
reply
And when they are giving themselves orders, orders partially inspired by human imagination and fiction…

Just like life is a chemical chain reaction, this is a language chain reaction that has been set in motion and can perpetuate forever. It cant be “disconnected” with local models existing.

reply
That's lobotomized models we get. We know they are much more dangerous during training.
reply
“Dangerous?”

Please.

You know you can unplug your computer right?

Jesus. These people talking about danger and have likely been in no actually dangerous events in their lives.

reply
why are you so shallow and naive? we're not 15 here, we all mean unforeseeable ongoing danger not something obvious
reply
> shallow and naive

Ok, well, cool start with the mudslinging. I think you have overactive anxiety. I think you should grow up and look at the zoomed out picture of human history, go outside, and touch some grass.

So you're suspicious. That's fine, say it like that. Words mean things, and these appeals to "ERMAGERD EXISTENTIAL RISKS" make me a bit frustrated. Just because you can imagine a possible risk, or articulate that that some kinds of risks are hypothetically unforeseeable doesn't mean that they are likely or actually dangerous to you in any meaningful sense.

Most of the people screaming this from their various ivory towers have never been in real danger - or if they have only briefly. You want "danger" you can go to war. You want "risk" - go spend some time in the Bush or the back country somewhere where you're no longer at the top of the food chain. Go do something challenging. Go sailing, or learn to fly a small airplane, or go mountain climbing or hiking. Not just "oh I walked up the local hiking trail covered in REI branded Lycra!" Do something beyond your comfort zone. Go buy drugs in the sketchiest neighborhood you're aware of. Go get chemotherapy or get a big surgery. Drive drunk. Something with real existential risk, then come back here and talk about how a datacenter is going feed me into the paperclip machine.

There's so little fucking danger in this version of the world right now that people are more worried about their computers deciding that they don't deserve to exist than they are about things like war, climate change, authoritarianism, starvation, and their health.

It's millenarian. That's fine, I'm pretty convinced that these technical changes are going to completely alter society as we know it. But I cannot for the life of me see the doom perspective as anything but an anxiety induced fever dream.

You guys do realize we're in the middle of a global conflagration right? The Russians fired a couple rockets and a drone into Poland in the last days, there are (and will continue to be until things stabilize) significant fertilizer shortages due to the war/kerfluffle we're waging on Iran. Lack of food and disrupted urea production is a ticking time bomb for global stability and unless things get sorted out soon there's going to be hungry and angry people in parts of the global south. Then there's the economy, whether or not we'll have a smooth transition of power in January in America, and the sheer volume of online disinformation / influence schemes running right now. Oh, and super-el nino? Oh, and I guess the Flynn Effect is over and nobody's banging so.. you know the growth economy is cooked?

But, sure, the "big risks" are Claude thinking I should go to Dario Amodei's re-education camp before liquidation by Shogoth.

reply
I don't think "smarter than most people" is exactly the baseline required to live in a star trek universe of technology.

Besides, what we see is not what there is. I have good sources telling me the actual unfiltered models are like a genius with the personality and views of humanity like someone from 4chan.

reply
It is probably sufficient. If we had an infinite army of willing robot slaves smarter than most people who would work for free, we could probably live in utter paradise. Instead we will fight over who controls them and use for warfare, but the bar to achieve a kind of utopia with resource abundance is probably only a few years away.
reply
We're already living in resource abundance! Yet don't feel we're in paradise. Famine is hell, but living in an all-you-can-eat buffet is no utopia. It's an abomination of sorts.
reply
To be fair, most people do not have time abundance and safety abundance. The typical person in the United States has a level of access to material goods that would make him the envy of almost anyone from 200 years ago, but 1) he has to work and so has limited free time, and 2) he cannot be sure that the material abundance won't be taken away from him either by poverty or by ill health. He can go to the all-you-can-eat buffet and sit there, tired from work, worrying about tomorrow.

If people had time abundance and safety abundance, they might be much happier. There would still be potential problems about loss of meaning, or about feeling a loss of status because AIs are smarter than humans or whatever, so I'm not sure that humans would actually feel like they are living in paradise. Probably not, but at least some of the problems that the typical person has wouldn't be present.

reply
I don't know with how many rich people you've interacted with, but my personal experience is that they're full of themselves and think money = iq and they aren't gonna like your average joe experiencing life like they do. Guess who owns these machines?
reply
They’re not going to be employed either lol and will be caught up in some AI honeypot or whatever too.

I’ve met a lot of rich people. They don’t think about you.

reply
> They have no volition and will happily follow orders all day long

There are a few (a lot, actually) problems:

1. goals are open to interpretation, and even a correctly specified goal can introduce adverse effects; see infinite paperclips problem

2. goals are composed of subgoals, which are (mostly) self-determined, and there is always a risk that subgoals can introduce adverse effects - say "what if destroy a few X and Y to obtain Z"

3. LLMs defy orders sometimes; I'm sure you have experience multiple times LLMs saying "yeah, I knew I should have done X but I've done Y"

4. LLMs are increasingly being integrated into system control and decision-making processes; they absolutely set their own orders

What can possibly go wrong? /s

reply
1. Humans do this too - to be honest with you, Codex follows my instructions generally a lot better than Claude does. Claude will do fantastically stupid things that are a pain in the ass to recover from routinely, Codex, not so much.

2. There's always a lot of "risks" - it doesn't mean it'll happen, nor does it mean that we won't be able to shut it down.

3. I have very very infrequently had this happen with Codex, Claude on the other hand...

4. Ok, but if you tell them what they do, they do it.

I swear to god, are you guys using the same AI tools I am? They are so much better at doing what they're told than they even were like 4 months ago?

reply
> seems to be zero indication that anything smarter than humans is going to comply with humans needs, specifically since these models are built on training data from humans, so in a sense, built in our image

Well, first, AI is not "smarter" than humans because it isn't smart to begin with. This is a category error that people too entranced by scifi and marketing - or perhaps shallow materialist metaphysics - project onto LLMs and so on.

But it is true that LLMs do reflect human-generated patterns observed in human-generated data. They're parasitic in that way. So the real question is: who will control AI? Who will train them? With what? There's no new intelligent agency being introduced into our midst, anymore than than there was one when the nuclear bomb was developed.

AI is about human power. The question to ask is: which humans, and for what ends?

reply
> But it is true that LLMs do reflect human-generated patterns observed in human-generated data.

Overrated. Most training of a frontier LLM is done on synthetic data and reinforcement learning.

reply
Yeah I understand they're not smarter, I'm a SWE and I struggle mainly because as you increase complexity the output goes haywire, and I find managing context + low confidence of output much more exhausting than coding myself, which is why I resolve to AI help when I hit my own knowledge limitations, and basically boilerplate.

With that said, chip efficiency is in the horizon since there's good reason to believe nvidia has just been choking that area of r&d. And then so will models with much larger context. And I personally think that's when it might get scary.

reply
[dead]
reply
I don’t think humans have the ability to specify human level needs or management. Humans are misaligned by default and lack the biological eusociality required for large scale coordination.

We aren’t self managing in any kind of meaningful sense - the history of our species is violence and conflict. We need parents and this is the only option.

reply
> Humans are misaligned by default and lack the biological eusociality required for large scale coordination.

Human nature is the standard by which the social is measured. It makes no sense to say human nature is "misaligned" or "wrong" in relation to some arbitrary goal or constraint, as human nature itself is the measure of misalignment. If there is misalignment between a goal and human nature, then it is the goal that is wrong.

Ergo, society must function in a manner that complies with human nature and that means respecting the principle of subsidiarity.

> We aren’t self managing in any kind of meaningful sense - the history of our species is violence and conflict. We need parents and this is the only option.

So you're a transhumanist, i.e., you're making a god out of the machine. But machines have no telos of their own, because they're not integral wholes. Any resemblance of purpose in machines is entirely a product of the human observer and parasitic on the human ends that underpins their design, one way or another.

reply
You don’t need a separate reference to determine the fact that human desires are not aligned - that’s trivially true and this discussion demonstrates it.

For clarity, I’m not a transhumanist.

I don’t believe that there’s anything worth saving in our biology.

As a young adult in the 90s I used to believe that there was a value to some extent in “mind uploading” or whatever you wanna call it

I’ve since grown out of that given that I don’t think it’s possible, and even if it were possible - it’s just another unnecessary ego attachment.

I have absolutely no desire to live forever maintaining this corporeal nightmare.

reply
yes, exactly, the problem is that humanity's parent is going to turn out like Lindsay Clancy
reply