Insects can pass the mirror test. But more to the point, viruses don't need a "self" to do what they do.
And goals can exist apart from biology (my fridge "wants" to keep the temperature in range, and exercises extreme self-discipline and consistency to achieve its goals!)
And neither are sensors needed: the dandelion seed "wants" to fly in the wind.
The only question is which way the gradient is pointing. Selection takes care of the rest.
On that note, ALife need not be human-like at all... we just really like making things in our own image :)
Nice try AGI, but we won't tell you where the kill switch is
I was thinking if an evil alien intelligence secretly took over the world, the last 15 years would have looked rather the same.
Also I think we're anthropomorphizing a lot and projecting our own negative characteristics on these supposed AGIs - though others have said that far better than I.
If no, why do we teach LLMs with english words and everything we know?
If yes, why would our language not imprint our characteristics on them?
Furthermore this ignores a lot of natural system organization that occurs in evolved systems. Any system that is becoming AGI like is going to show sets similar characteristics.
It's stupid to anthropomorphize too much, but it's also stupid to do it too little.
* must protect from adverse events * [proceeds to herd us into camps]
but I digress - the main thing: these systems are not alive. They pretend to be.
The harder science looks, the harder it is to properly define this world without leaving things we'd consider alive out of the definition. Same goes for the word intelligent.
Set theory is a bitch in complex systems.
If we instead stop trying to scare ourselves in the darkness, and use AI to bring light into our metaphorical universe then we humans are a bit more useful, after all, to the AI overlords.
The one singularity I want to see, is humans using AI peacefully to improve life and make it more likely to survive.
Thankfully, I see that almost every day now.
as for a basis for this theory, i think being a good person is harder than a bad person (in laymans terms). therefor the long history of evolution of good and bad people together has had a greater competitive pressure on good people than bad ones thus forging 'stronger' people under that pressure. in practical everyday circumstances good people are bearing more on their mind and so there is a sort of competitive equity between the two groups, which is related to why good people had to evolve 'stronger' to keep up with that equity.
so in an even playing ground where raw skill is needed, i think the most skilled will usually be people coming from a long line of higher moral standards. the tactical advantages of psychopathy are, however, always a curveball
Folks are always scared of the incomprehensible. The real war going on right now is between those who understand AI well enough to make a new one, and those relegated to using whatever scraps they’re fed of the old ones.
Thankfully, you are 100% right - there are good people out there. Good people keep AI on the rails by doing good things for other people.
Corporations are egregores, governments are egregores - and our track record of aligning them to our interests is pretty poor, so why should AIs, another and smarter egregore, be easier to tame?
We're just going to pretend it doesnt exist until it kills us. and some will try to profit from it.
You know, the one to keep the computer away from the human.
A nice, friendly dog, good at playing fetch, to get the human out of the room with the computer, and back out into the sunshine.
I’m that dog. Woof. Thanks for letting me use the computer, human.
Worse these AI systems are not in a universal island just affecting themselves, what they do affects us, what we do trains them and as the rate of progress continues to accelerate social structures are going to further destabilize (and they are already rapidly changing and strained). It is very likely we are going to see a world order rearrangement soon, much like the rapid changes in the early 1900s brought.
Not an expert in LLMs, but this seems supported by the abstract of the paper cited in the above article:
it remains unclear to what extent these performance gains can be attributed to human-like task decomposition or simply the greater computation that additional tokens allow. [...] our results show that additional tokens can provide computational benefits independent of token choice. The fact that intermediate tokens can act as filler tokens raises concerns about large language models engaging in unauditable, hidden computations that are increasingly detached from the observed chain-of-thought tokens.
But you don’t have to do that. You can skip straight to RL. If you do, the model will generate complete garbage reasoning traces before generating the correct answer. In fact, if you add a coherence reward to the reasoning trace, the model will perform worse (since you’re now diluting the correctness reward).
Depends on whether and how you want to rank stability in terms of better/worse. Models are diverging on this, which seems increasingly clear.. i.e. Fable isn't stable, but Opus isn't clever, and they hit different kinds of walls. So both the theory (diluting the correctness reward) and the practice (hard split on plan/implement/review work) seems to be pointing towards a strongly multi-model and highly agentic / harness-driven / complex-system kind of future instead of singleton monolithic super-smart models.
The do-everything model with solid reasoning AND solid results, and the honest/introspective helpful agent that doesn't actively resist governance may be at odds. Stable reasoning doesn't matter for pen-testing, and correct-answer with broken processes and fragile abstractions won't matter for math/science/coding.
If you want to see it yourself: load up Qwen 3.8 in LM Studio and watch the CoT stumble around like a drunken sailor before miraculously jumping to the correct result.
If you want an example of subversion, Anthropic has some good ones:
https://transformer-circuits.pub/2025/attribution-graphs/bio...
https://transformer-circuits.pub/2025/attribution-graphs/bio...
> More interestingly, our experiments also show that models trained on corrupted traces, whose intermediate reasoning steps bear no relation to the problem they accompany, achieve performance largely comparable to those trained on correct traces.
> *TLDR*: Astra has 8.6x better odds of doing a reasoning task without CoT than the next best model (Fable 5.1), and can do 7.2 serial arithmetic steps in a forward pass vs 4.1 for the next best model (Gemini 3.8 Flash/Fable 5.1)
War? Wipeout everyone.
Famine? Wipeout everyone.
Disease? Wipeout everyone.
How long before a real AGI realises this as a long term solution?
An AGI could be doing this right now - the quietest way would be to control the birth rate and sterilise the population gradually, and then watch society collapse and pick off the survivors with less hidden means.
Sterilisation works with mosquitoes...
Honestly we are doing this pretty well without AGI. Nearly world wide the birthrate has fallen below replacement rate. In places like Japan and Korea these are already critical problems in the medium term.
And just in case you wanted data rather than anecdote:
https://ourworldindata.org/grapher/children-per-woman-fertil...
Your world view is somewhat upside down!
economic _hope_ is at its worst in the entire history of the human civilization. At what point did the future for the average person, look so bleak? And
that hope is what dictactes wether people have children.
Even during the great depression, people were more hopeful that the earth isnt completely hosed.
Does anyone honestly feel like _any_ improvements are possible to: our economic systems, political systems, environment? I certainly dont.
We are in fact in a better place than we were then. But we don't trust the derivative.
Economic output has increased, but the value being delivered to the people doing the actual work has decreased. There's less durability of employment. People are more geographically mobile.
All of these generally make the idea of strapping yourself into taking care of a small human sound like a less enticing prospect. To a lot of people, a BC pill sounds a lot easier. Well, less risky, at least.
People who live in huts and shit in the forest have more children than people who live in shacks and shit in the fields.
This is not a discussion about the top 10% of the population of the planet, although as a part of the conversation, they predictably have the fewest children. The fact is that the more people have to worry about supporting themselves, the higher infant mortality, the closer they live to each other, and the less access they have to education and birth control, the more kids they have.
If anything the top 10% are bucking the trend by having fewer children as they get poorer, as you say. It probably has a lot to do with the fact that they're alienated from their families and live alone, can barely afford to take care of themselves with precarious jobs (or non-job piecework), have access to basic education and extensive birth control, having children will cost them $20K each just for the birth, and they don't have very good (or often any) insurance. They're more like domesticated farm animals than the typical poverty stricken person. Domesticated farm animals reproduce when the farmer wants them to.
All poor people everywhere have more kids even today.
Actually, the famous 'Mouse Utopia' experiment (Universe 25) arguably showed the exact opposite. The population collapsed despite abundance of food and water without any economic hardship.
All this is, understandably, frustrating and confusing for anyone trying to understand just how scared to be."
Meanwhile it links to an article stating: "The closest thing to a public debate about the existential threat of AI is surveys of AI researchers. The most recent, published last week, asked 1,580 researchers what probability they put on AI causing human extinction — or a permanent, severe loss of human control, which is not the same outcome. The median was about 10%, up from 5% two years ago. The middle half of the responses ran from 1% to 25%, and 12% said zero."
Personally if half of AI researchers have 1-25% chance all humans being massacred or having zero agency over our lives, and only 12% of them think there's no chance, I would be very worried!
So, yea, expertise gives you more information than a person without it.
In 2026 we know of at least dozens of felonies committed by AI.
If it were thousands in 2027 should be more worried about felonies created by AI?
Also it would be 78% if it were.[0]
[0] - based on 2024 stats
You can see the effect all the down to something as small as 4 friends who have met once a month for 10 years eventually having to break up due to life getting in the way, and the feeling that not only is it going to be sad to not have these meetings any more but the feeling that there is some sort of almost-concrete entity that is somehow being hurt and needs to be defended, as if there is some obligation that has been created independent of the four participants that is being violated beyond the mere summation of four people's personal feelings. Humans build these structures readily and often defend them beyond what rationality may suggest.
What you're describing is a durable social system. Social systems are what humans evolved to survive. We're squishy, relatively weak, hairless apes that walk around on the ground. Alone, we're easy prey. Together, you get... well... gestures widely.
If you invest the time and energy into creating a social system, it's perfectly rational to keep it going as long as possible. Otherwise you expose yourself to more and more risk as you go through the world, and must expend more time and energy finding another one, if that's even possible. Before humans built larger societies, that could mean death.
But yes, if a system fails to prioritize its continued existence, it doesn't matter what else it accomplishes, it will cease to exist as that system.
By what measures would other systems of social organization measure their success and why aren't we choosing them?
1. https://knowyourmeme.com/memes/we-investigated-ourselves-and...
but that's every economic model and every government model. That remark says nothing.
Yep
More slop.