It sounds more like "everybody I know thinks AI is the top concern" when everybody they know is people like them, i.e. an echo chamber. It gives me no confidence in any other claims.
AGI and AI doom scenarios have been evergreen topics in rationalism for years. Long before we had enough compute power to do anything at all useful with generative AI. Many of the rationalist scenarios about future AIs had very religious themes where the AI became so powerful that it was effectively a god.
Rationalists had concept's like Roko's Basilisk, a thought experiment where a future all-powerful AI might choose to punish anyone who predicted the arrival of AI but failed to contribute to the arrival of AI.
The rationalist communities took this thought experiment so seriously that the biggest rationalist forum of the time formally banned discussion of Roko's Basilisk for 5 years. They believed it was an "info hazard" because once you knew about this risk, you either had to contribute to bringing about this future AI god or you risked being punished by it in the future.
If this is new to you, you might think I'm exaggerating or making things up to make a group look bad, but this was such a core belief that it has its own Wikipedia page: https://en.wikipedia.org/wiki/Roko%27s_basilisk
The people writing AI 2027 and AI 2040 were immersed in these cultures. That's why they think that AI is the primary topic that anyone is talking about in elections, just like the Bitcoin people all thought crypto was the only relevant election topic.
The GP comment is getting attacked, but there really is a heavy religious fervor background to the AI fears in the rationalist community. The religious angle and extremism of the forums is downplayed when in crosses over into mainstream topics, but it's been there from the start.
Oh, I didn't realize it was 2028 already!
For me its as simple as watching how people talk, and seeing how in every single case whatever the next thing is, if you believe It, there is only ever justification of doubling down, doing more, going deeper, reducing any doubt. These are not scientists, they're business people and salespeople, and a few optimists having recently on paper solved all their worldly financial needs.
Even if one throws that aside, spending time exploring and building with the most state of the art LLMs is just as instructive. I'm watching the implementation - whats working is ML models trained on specific domains (not much different than 5+ years ago), and whats not working is a general model that humanity can let go to work on its own. Sit in front and observe ideas turn to the samey intellectual, high-syllable mush. Its productive, but not in any way that's promised.
Important point. LLMs were early on hailed as the first general-puprose AIs that can perform any task (remember "Sparks of AGI"?). Today they're increasingly promoted for specialised applications - coding, as a for instance.
As usual, AI skeptics are moving goal posts. Modern LLMs are on a completely different level in terms of how GENERAL they are vs anything pre-LLM. You can give it a completely novel puzzle and it will solve it. 5+ years ago you had to train NN to solve particular type of puzzle.
They're proposing an alternative, which is a global brake on frontier AI research to keep the basilisk in its jar until we work out what we're dealing with and how to handle it.
> AI risk is string theory for computer programmers. It's fun to think about, interesting, and completely inaccessible to experiment given our current technology. You can build crystal palaces of thought, working from first principles, then climb up inside them and pull the ladder up behind you. People who can reach preposterous conclusions from a long chain of abstract reasoning, and feel confident in their truth, are the wrong people to be running a culture.
I understand how people running in the same scene fall into the echo chamber effect and get gulped into the cult, but why does everybody want to be a prophet?
Its not everyone building crystal palaces in their mind, they're all building fortresses. And they can't be wrong in their fortress or it breaks their world view which they cannot accept.
So the goalposts will be moved whenever necessary in that case?
No amount of incredible advances in AI will ever get skeptical HN commenters to take AI's implications seriously?
"The Gish gallops will continue until the nagging doubts have been silenced"
If you can’t provide a realistic path to achieve something, you’re asking people to believe in science fiction.
You could tell me that a rock’s molecules are comprised of protons, neutrons, and electrons. Blood is also entirely protons, neutrons, and electrons; so theoretically, one could rearrange stone into blood. But without an actual method to do so, it sounds like you’re telling me that you can squeeze blood from a stone.
> the human brain exists, it is not made of magic, it can reproduced
Yeah. It only takes 9 months and ~18 years of training…
> But saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Let’s be clear. Everyone is talking about silicon transistors here. That’s what we’ve got.
Digital computers have real limits. Sensors and other sources of training data have real limitations. It’s not clear that we can organize them in a way to reproduce organic brains.
Like, afaict, for many on HN going from ELIZA->Fable 5 just didn't cause any update to priors regarding this whole philosophical question. The argument against has remained unchanged. I don't see any point in arguing about it, I just find it very strange.
One popular idea is that these systems will asymptotically approximate human intelligence because they're trained on mostly human-written texts. Not only is that untrue, it's also directly contradicted by our experience with previous RL-systems, where they seem to breeze right by human ability without even the slightest hiccup.
Most human systems are much, much, much more complicated than most closed world games (which is where RL approaches have seen massive success, mostly through self-play).
Like LLMs are great, but I honestly can't see us getting actual general intelligence out of them.
Can you elaborate on this? I am clearly not aware of this line of thinking and the related contradiction.
What priors should be updated?
Yet it is generating billions in revenue which Eliza did not.
Perhaps all we need is scale and some refinement techniques to eat a big fraction of the economy.
If unimpressive inputs lead to impressive outputs, that should make you more worried, not less.
It can hold many complex and partially contradictory thoughts in its head at once, in a way that feels significantly superior to Opus (for example). And then can make reasonable syntheses across these.
In a couple rounds of back and forth, with relatively low effort (but strategic) prompting, it produces complex, accurate analyses in 5-10 minutes that would take me multiple hours of hard, very focused work.
I still need to remain tightly in the loop, providing frequent course correction, clarification, high level reframing, nudging, and grounding.
It incorporates my feedback incredibly well.
It’s honestly staggering. Fable has changed my assessment of the current trajectory more than any model since possibly gpt-4. Opus 4.5 of last year might be a close second.
———
My advice for anyone who wants to get more value out of these tools:
When a model does something idiotic, don’t throw your hands up in the air. Be curious. Try to turn it into a puzzle to be solved.
It know it’s hard sometimes, especially if you are drowning in slop from other people… or generated by yourself, heh.
It can be exhausting. I struggle with this also. I have thoughts on how to make it better. We shall see.
As for "updating priors", that goes both ways. There's plenty more reason to think "hey, transformers and RLHF might actually make some killer products" but certainly no reason to think the few people who didn't realise that "GPT3 is too dangerous to release" and "all software engineers will be replaced within 6-12 months" were marketing rather than prophecy have some kind of special insight into how it's all going to pan out. Clock's ticking to the promised 2027 reckoning too...
Now write down a blueprint for superintelligence.
So I've given you two impossible engineering challenges, but one of them is feasible in principle because we at least have the tools to begin to tackle the theoretical calculations and therefore we can do engineering. We cannot do engineering on the superintelligence problem yet.
In my view it would be insane to believe we can build something that we can't even reliably imagine yet.
However, if you had had demanded someone for a blueprint in 1925 of how to design such a magic bullet, especially a magic bullet that targeted virtually all forms of bacteria, it would have sounded ludicrous. Yet, 20 years later, the world was manufacturing 6-7 trillion units of penicillin a year, capable of treating 3-6 million people. And that’s in spite of the fact that Fleming’s work sat mostly untouched for a decade before Howard Florey and Ernst Chain seriously set about to isolate and purify the substance.
You can quibble and say that penicillin was discovered, not designed, which is certainly true. But I would ask you to consider, does current AI development look more like design or discovery? Does it look more like analytical engineering or evolutionary selection? I would say on both counts the latter, in which case, we should prepare to be surprised how long it might take to make revolutionary advances. And that’s on both sides of the ledger, we might find ourselves stuck in the current paradigm for a long time. But, we might not be.
I wouldn't bet on evolving an intelligent, sentient being-in-a-box on a computer any time soon though. I'm of course prepared to be pleasantly surprised.
That said, I think it's pretty clear that LLMs are not going to get us there.
This is obviously complicated by the fact that LLMs/Agents are useful by themselves, but that’s not really the topic at hand.
This is why biological comparisons are weak, we talk about a few agents verifying and checking LLMs, meanwhile the world consists of almost an infinite number of the same, just operating on different time scales. I agree that with we don’t know the timescale, and we definitely don’t know if long term it will continue to work ”adding more of the same”. Throwing more penicillin at the problem sure as hell didn’t, but it looked great initially. And I’m obviously not arguing the human benefits of penicillin, just that what we thought would work forever quickly didn’t.
> Life, uh, finds a way.
Imagining something in advance is not necessary at all for scientific advancement. This is particularily true in AI, and no one expects to imagine what superintelligence is until after it is created. You set up your datasets, your architecture tweaks, and measure the results on some set of benchmarks. There never was a blueprint, no plan beyond the experiment itself. We're not even close to understanding the things we have already created, and yet we created them. So why expect anything else for the next step?
That is simply not accurate. There are examples of scifi novels, novellas and other media that dealt with it. We can argue over whether it was that exact format, implementation and so on, but that 'shape' ( to use a common llm term ) of technological advances was very much explored.
Then why does anyone expect to create it? I'll take a stab at an answer: they think an LLM is some kind of "incremental improvement" and therefore a step along the inevitable path to discovering AI. But that seems delusional to me. I can't imagine anyone sound of mind who knows how an LLM works thinks it's actually intelligent. So in what sense is it an "advancement" on the path to AI?
The concept of an incremental improvement in an objectiveless search in a high dimensional space is.. absurd.
It's reasonable to doubt that LLMs are a path to AGI, but I don't understand how this is still a matter of dispute in 2026. What's your definition of intelligence that doesn't cover an entity that can translate fluently between dozens of languages and also solve open problems in mathematics? And be real-if you have one, is it a definition you or anyone would have given a decade ago, or are we doing "god of the gaps"?
That's more or less looking for interesting patterns in a jpeg or another lossy compression result. It's interesting that the models seem to be able to (fairly) reliably return relevant chunks of the image. Even more interestingly, they seem to be able to invent plausible chunks of image that aren't even there. That doesn't meet my bar for intelligence though. I'd need to see it learn and adapt. I'd need to see it be clever, not merely "knowledgeable". I'd need to see it capably analyze itself. I'd need to see it reasonably estimate uncertainty and know itself in the sense that it has some idea how right or wrong it is about something. I'd need to see it exercise judgment.
I don't think I'd give a different answer a decade ago but who knows.
[edit] For all we know, one of the salient features of intelligence is that intelligent beings are incapable of precisely defining it. I'm not sure how productive it is to attempt to do so.
Actually, stronger - it's valid in some circumstances to say something is infeasible to precisely to define and you'll just know it when you see it. But I don't think it's reasonable to take that stance and then assert that "anyone sound of mind who knows how an LLM works" must agree with what you see. You gotta pick between striving for rigor and denying your opponents' soundness of mind.
Except it did none of those things, really, because that's not how it works. This might help, it's a good writeup: https://www.0xkato.xyz/how-llms-actually-work/
We know how these machines work, it's not mysterious, there's nothing "extra" happening.
> We know how these machines work, it's not mysterious, there's nothing "extra" happening.
It sounds like you're saying "We don't know how brains work, they're mysterious, there's something 'extra' happening", and using that as justification for why you're saying a computer, an AI, can't "understand".
I think most people on Hackernews now who would use the phrase "my AI worked overnight and hypothesized, compared, etc..." already know how an LLM works, and still chooses to use those words. So the issue isn't that they don't understand. It's that they understand and still use those words. So the disagreement is somewhere else.
OTOH we do know how neural nets work, and they definitely don't do "thinking" or "reasoning".
Also, not to get too reductionist about this, but what do you posit is special about what is happening when humans think? Intelligence is hard to define so clearly, I reckon.
No, it does not imply that at all. Google "temperature in LLMs".
> what do you posit is special about what is happening when humans think?
I don't. And IIUC nobody knows, but I'm not a brain scientist. There have been some wild theories over the years (recall Penrose's). I don't really have a dog in the hunt, except that probably whatever is happening is physical. It doesn't really matter, except insofar as whatever is happening very probably isn't what LLMs are doing. We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
No need to condescend, I'm very aware of what temperature is for LLMs. But I'm going to push back - if you're claiming all LLMs simply do is a stochastic _search_, how can that produce novelty, in the conceptual sense? (I'm not, for example, talking about novel rearrangement of existing ideas and code)
> We know enough about what an LLM does, and what a brain does, to be quite certain they don't work the same.
I don't think the claim is that LLMs do what brains do - I think the correct form of the counterargument is that _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
By reaching into the voids of its embedding space and returning tokens related to nonexistent semantics. Or, if you like, "hallucinating". The hallucinations which are useful we might call "novel".
> _whatever LLMs seem to be doing_ produces end results that were previously only possible through the application of human intelligence, so there must be some axis of however you define human intelligence that LLMs currently seem to display as an emergent behaviour.
I don't think that has earned its therefore. Another perfectly reasonable explanation is that LLM's output is a close enough facsimile to intelligence that if you allow yourself you can easily be fooled into thinking its intelligent. That's not the same category of thing. It's not an incremental step away from intelligence. It's a whole different animal.
This sounds to me like an admission that LLMs are not just doing a stochastic search, then.
> close enough facsimile to intelligence
What's the distinguishing criteria then? How can you tell the difference?
- Something contained in the data set, not necessarily the same thing for every iteration of a given query
- Something not contained the data set (hallucination), not necessarily the same thing for every iteration of a given query
Does that clear it up?
> What's the distinguishing criteria then? How can you tell the difference?
All the ways they fail to exhibit intelligence. They can't learn. They can't adapt. They can't reason abstractly. They can't count. Etc...
I find the rebuttals pretty convincing - that there seems to be some emergent behaviour that is not simply just next-token-prediction, or that the ability to do accurate next-token-prediction requires something "extra" that LLMs have.
> All the ways they fail to exhibit intelligence
Another implicit admission that there _are_ ways that LLMs exhibit intelligence?
The next step then would be to design and conduct experiments that isolate this effect. Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining. Isolate it or distill it so it can be studied directly. Until then, it's easiest to dismiss it as imaginary.
And you're happy that the replication of LLMs across many foundation model companies is insufficiently reliable?
> just something you're imagining
So the alternative explanation you're suggesting to emergent LLM behaviour is mass independently-corroborated human hallucination. Which is more likely?
Also it really does seem like you've moved the goalposts a lot here without really giving me a substantive response.
This is not at all what I was saying. I think you've already conceded that LLMs demonstrate emergent behaviour but you dismissed it as a "close enough facsimile to intelligence". I was saying that the emergent behaviour is reliably replicable, in response to your following statement:
> Figure out how to make it happen reliably and in such a way that you know it's actually happening as opposed to just something you're imagining.
I think there is real work underway in the area of interpretability. In the meantime, there appears to be plenty of empirical evidence for the claim that LLMs exhibit some sort "intelligence" in the enormous penetration that agentic coding has achieved in software development? Do you deny the usefulness of LLMs here, or are you going to assert that actually software development requires no intelligence of any sort?
No. Please don't put words in my mouth. What I said is that an LLM compresses a bunch of information into a semantic embedding space and then does sort of a stochastic search in that embedding space. Any similarity to "intelligence" is accidental. You may look at the results of that process and "see" thinking or reasoning or something, but it ain't there.
> "intelligence" ... agentic ... usefulness
I don't think LLMs need to be intelligent to be (at least narrowly) useful. No more than random forests or genetic algorithms do at least.
[edit] Look, this has devolved to the point where it's no longer productive to continue. If you're going to state things like this as fact, there's really nothing more I can do here:
> emergent behaviour is reliably replicable
Go collect your Nobel prize then! This is no longer a discussion grounded in reality.
On what grounds? I don't think you've provided any evidence other than LLMs can't "adapt" or "learn" to show that LLMs do not show intelligence in any way. I think it's clear that there must be some emergent form of intelligence over words from just the agentic coding ability alone. I am not claiming that LLMs are intelligent, only that they display aspects of what we understand as intelligence.
> I don't think LLMs need to be intelligent to be (at least narrowly) useful
I agree! But they are more than narrowly useful, and they absolutely do not belong in the same category as random forests or genetic algorithms!
> Go collect your Nobel prize then! This is no longer a discussion grounded in reality
Once again you are being condescending while misrepresenting my position. The emergent aspects of "intelligence" have been replicated by virtue of independent LLM vendors training their own models - I am not making a stronger claim, you have misunderstood me.
Thanks for participating.
It doesn't seem like you're being consistent here. I'm concerned there might be some motivated cognition going on.
"What is true is already so. Owning up to it doesn't make it worse. Not being open about it doesn't make it go away. And because it's true, it is what is there to be interacted with. Anything untrue isn't there to be lived. People can stand what is true, for they are already enduring it."
Lol. This is more telling about your implicit unscientific preconceptions that you wanted to reveal. Of course there isn't anything "extra". Where do you think intelligence comes from, some mysterious realm? It's physical, computational. The fact that at the bottom we produced it via matrix multiplication is irrelevant. Maybe humbling. You are denying a visible fact (a machine performs tasks that require flexible analytical and cognitive skills) precisely because there is no magic happening anywhere.
Well, no. I don't think it comes from some mysterious realm. I think that which is not physical does not exist [edit: and if you like I'll follow that one right down the rabbit hole--continuity and infinity are useful delusions]. But that eminently does not mean we know what intelligence is, let alone how to build one.
> The fact that at the bottom we produced it via matrix multiplication is irrelevant.
Huh? We don't even know what "it" is. How can you say you produced it?
> a machine performs tasks that require flexible analytical and cognitive skills
You see that, I see a lucky stochastic search result. Don't underestimate the "creativity" of random algorithms! They can do some wild shit! This is nothing new, we've been playing with these toys for like 70 fucking years. It's only recently that they started spewing words and everyone lost their minds over it.
> I see a lucky stochastic search result
Again you're reaching for a mechanistic explanation of some kind (let's leave for the moment whether it makes sense or not) as if having an explanation somehow contradicted a display of intelligence. It doesn't. Yes of course we made it, we know how it works (ar some level) and there is no magic. But what matters is the result- this machine, matrix multiplier, stochastic parrot, consistently displays intelligence, to the point of being able to perform very complex, open-ended tasks that integrate discovery, planning, tool usage, decision and even some aesthetic sense, understanding and using natural language, context awareness, you name it.
> This is nothing new, we've been playing with these toys for like 70 fucking years
Lol no. For god's sake. Hundreds of billions of parameters organised in a specific architecture and trained with unimaginable amounts of data and compute? Unless by "these toys" you mean "any computer program vaguely AI-related".
IDK, it doesn't seem like they actually do any of that. To me it seems like they have good enough semantic embeddings that they can kind of approximate those things, sometimes, well enough if you don't look too hard. This is enough to fool people. Of course there's gold in them hills--some recent mathematical results were found there. But to say that's "intellgence" is to say that lossy compression is intelligence. It's static. It does not learn. It does not adapt.
> Unless by "these toys" you mean "any computer program vaguely AI-related".
Not "vaguely AI related". I mean stochastic computer programs that can do things that look awful thinky. They've existed for a long time, but only recently (due to word2vec and other advances) have the results been words that mostly go together well instead of numbers. For some reason people seem to think a lot less critically when the output is words. IDGI but it's a whole thing.
Uhuh. I really shouldn’t be replying to this type of comment from a throwaway.
But the extremely powerful semantic search that we get from LLMs isn’t enough. I don’t think anyone is credibly arguing otherwise?
Agents already are a layer on top trying to bridge the gap. But they’re really just using LLMs as a heuristic to explore extremely NP problem spaces. The notable successes with agents so far are when we can provide them with a solid verifier and preferably additional context hints on the steps to take in the problem space. See the test oracle problem on where this gets us.
So forgive me if I think that it would be enough of a jump in computational complexity to remove those guard rails that it’s not feasible. But don’t say that I’m clueless, stubborn, or confidently wrong.
That's not "irrelevant", it's fundamental.
One can simultaneously believe AGI is possible, be only modestly sceptical that our current methods are likely to yield it in the near term and still find the religious ferocity enveloping its discussion silly.
> saying that the human brain cognitive capabilities cannot be reproduced on other types of substrates is stupid at this point
Straw man. Nobody argued this. The discussion is around how urgent it is to policy treat a future hypothetical.
Not that I'm complaining. Cynicism is the failure mode I rely on HN for. It's the populism that's been getting to me.
Fair enough. I didn’t see anything novel in the article. So treating it as a motif within the abovequoted “Superintelligence: The Idea That Eats Smart People” context is fair and a real argument.
> Cynicism
Cynicism isn’t the opposite of blind optimism. Nihilism is. I’m not seeing a rejection of the article as being baseless as cynical or nihilist. It’s just pointing out a cultural thread that doesn’t seem to be useful.
Cynicism is defined as
>An attitude of scornful or jaded negativity, especially a general distrust of the integrity or professed motives of others.
I'm not saying that cynicism is automatically wrong, just that I once could trust that, when HN is wrong, it is due to cynicism applied in excess.
We don’t know. Which makes proposing rules around it based on fiction more than science silly.
Perhaps Anthropic will create God in the Machine. Not foreclosing on that. But will it matter so much who was fucking around with Opus five e-folding times ago?
Either ClauDeus is benevolent and lifts you up (not left behind) or it isn’t, or not to you, and you are culled by a drone (left behind regardless).
Serenity Prayer time.
This is an objectively wrong opinion.
> Yall, with your incredible wealth and resources you could do real good in this world and make society better, healthier, better educated, and the whole world more equal, just, and reduce the desperation and suffering. Reject the false and self-serving narratives that empathy doesn't matter, that altruism isn't "effective". You can change a person's whole life in a moment.
Confused at who this is directed towards. I'm fairly certain that the article was written by people who (at some point) identified as effective altruists, most of whom would enthusiastically agree with this. This community didn't start as AI researchers and later choose effective altruism; they were effective altruists who chose AI safety research as the most effective way to improve the world. Given that you apparently share their goals (a better world,) isn't it worth at least hearing them out on their methods?
Their methods are about convincing others that things that enrich and empower themselves at the expense of others is "improving the world". This isn't the stance of serious people who want to improve the world.
I'm fairly certain the authors would be happy to see AI shut down indefinitely. They just don't believe that the coordination problem is solvable. This is their best attempt to come up with something workable in the real world, or at least get people started thinking about it.
As to a better world or super intelligence, I’ll believe it may be possible when I see some signs of intelligence from what people are calling AI, instead of plausible text and image generation based on a very large corpus.
Altman says he thinks it is an "incredibly flawed movement"
https://x.com/sama/status/1593046526284410880
The dislike is mutual. Here's a long video takedown of Sam from a major EA org:
I agree that there seems to be a huge problem with value drift--either people who used to research AI safety, pivoting to building AI (looking at Anthropic) or people who only ever paid lip service to safety (I tend to put Musk and Altman in this category.) These people need to be held accountable, but it doesn't mean every AI safety researcher ever was a stooge or a fraud.
The point was I think pretty much everyone else saw the bait and switch coming...
ChatGPT was announced three and a half years ago, 30-11-2022.
https://hn.algolia.com/?dateEnd=1594339200&dateRange=custom&...
From your bio I suspect you're already in the cult.
Who is this supposed to be arguing with? It sort of reads like it's trying to disparage "effective altruism", but I'm not sure.
Setting aside any of the AI stuff, I've started to find it pretty grating when people seem to imply that transferring millions of dollars from wealthy people in California and the UK to impoverished Kenyans and Rwandans, or buying malaria bednets which can save a child's life for the cost of a fancy new gaming rig, is "self-serving" or something because weirdos are doing it, while true caring for other people involves [unspecified thing that doesn't appear to ask any material sacrifice comparable to donating a large percentage of income].
You can change more people’s lives, more substantially, if you donate effectively. Effective altruism started out as (and the majority of effective altruist financing is committed to) an effort to rationalize what has historically been a very emotionally driven activity by deploying insights from developmental economics. If you want to take longtermists to task, go right ahead, but please refrain from torching anti-malarial or child vaccination programs while doing so.
OpenPhil changing its name to Coefficient Giving, 80000 hours and bluedot and (to a lesser extent) CFAR dropping other initiatives and switching to AGI promotion… to my knowledge GiveWell is the only other big name that continues to advance other initiatives. Then look at figureheads like SBF committing fraud and begging for a pardon from the architects of the USAID shutdown… We begin to paint a picture of a community that’s (by and large) abandoned its principles for power.
I know the view from the inside is more nuanced, but I think it’s a reasonable association for random members of the public to make.
My critique of the EA community is that it’s myopic and unregularized. If you really think AGI is make-or-break for civilization, it’s completely rational to deprioritize side bets.
I’d be curious to hear you expand on this. What binds the EA community together, from the shrimp welfare enthusiasts and wild animal initiative, to the longtermist lightcone obsessive, to the people funding vitamin A supplementation, is simply a commitment to maximizing the number of quality adjusted life years saved each year and a belief that empirical observation can be used to improve that number.
To my mind, this is a valuable insight on its own. Yes, if you come to such a heuristic with absurd prior beliefs, such as whether 100k neurons alone have QALYs in the first place or by placing equal value on people actually alive today and hypothetical people in the far flung future, you will get absurd results. Garbage in, garbage out. But that’s not an indictment of the fundamental insight, especially when you consider how poorly allocated the roughly $2 trillion in global charitable spending is.
1. Stable housing
2. Access to safe drinking water
3. Access to food
4. Access to healthcare
5. Access to education
6. Stable governments
The EA community has so many "ideas" about what would help, when all they need to do is focus on those six and the world would be as close to a utopia as you or I could hope to see in our lifetimes.
I legitimately thought you were kidding about the shrimp welfare initiative, but after looking it up I was more infuriated about EA than I normally am when it comes up. I can think of several causes which would be better served with 3 million USD, and all of them take care of human beings. Living, breathing, intelligent human beings. These people should not be in any position of power or taken seriously ever.
The existence of mythology describing Scenario X is not a valid argument against the plausibility of Scenario X.
If we can acknowledge the possibility of nuclear doomsday without running an RCT of sample size 100 Earths, 50 of which undergo nuclear Armageddon, to verify that nuclear holocaust indeed a real phenomenon... then we can do the same for AI. Understand the arguments being made instead of engaging in these guilt-by-association arguments.
Modern AI capabilities are already mind-boggling by the standards of 20 years ago. We should at least prepare for the possibility that trends continue on the current trajectory.
https://en.wikipedia.org/wiki/List_of_dates_predicted_for_ap...
Notice the many times when a prediction failed and the so called prophet would come back in a few years and give you a new date. And they would tell you, well, this time it's real.
I do find it ironic that many of the AI predictions are coming from the self titled "rationalists." It seems like building your identity around being rational and immune to psychological pitfalls is a good way to ensure that you don't even notice that you have walked straight into the one psychological trap every cult has employed since time immemorial.
Then, you opened the page and read it and realized that this prediction was contingent? You know what a conditional is (I would assume) if this, than that?
Then you realized that the only reason you were posting this comment was as a sort of silly gotcha "Oh look at the guys who keep increasing the number" instead of talking about the differences between the scenarios?
Then what happened?
You actually have to look at the substance of the prediction. Sorry.
The one thing I will say that they are correct about is that AI does have the potential to be highly destabilizing geopolitically, even if they get everything downstream of that wrong.
But instead of weird religious deities and practices, they've wrapped up their true believer zealotry in some kind of mishmash of "AGI is coming real soon now" like some kind of manifest destiny.
You could probably put people in FMRI machines and ask them to give a 30 minute lecture on the topic of AI and find that the same parts of the brain are activated.
https://www.cnn.com/2016/11/29/health/religious-brain-mormon...
They are the crowd who need to understand your framing the most, but they’re completely shutdown by your framing, as any religious follower would be.
Your message gives me hope that not everyone’s drunk on the kool aid.
> Plan A is primarily a recommendation, not a prediction.
That sounds nothing like “religious fervor”
The fact that technology can increase existential risk for civilization is not fantasy. It’s a risk that should be reasonably discussed.