People that previously have barely any experience in tech, now being hired in AI startups because they are good bullshitters.
Oh, design too. I’ve met many engineers who simply don’t see any value in design.
Of course this should be differentiated from a professional setting.
And under no circumstances should we put that stuff into the knowledge bases we base our "AI" on without making sure it takes into account the context.
Yes, and that is the problem. It used to be if a product looked polished it was fairly polished engineering wise as well if we compare to todays AI slop. You can see that on steam, before AI slop games a game that looked polished mostly worked. But today you can get a game that looks like they put in a lot of effort, but its all AI slop and everything is flaky and broken, I've had way more such experiences the last few months than before that.
A human coder that was capable of coding a complex game typically also was good enough and tested things to make most things work. There were bugs etc, but at least buttons did things.
Obviously.
Can they do the job? Because right now, government decisions are based on AI generated code, which was verified by nobody who can do that. So the cost of an unsatisfactory answer is quite high.
I like the symmetry of people being unable to detect "skill" in people who are great at manipulating language and in LLMs who are also great at manipulating language.
The sort of person that's going to offload their thinking to AI is the exact sort of person that is not going to verify anything because they've already offloaded their thinking to AI.
B...b...but the Anthropic trainer said we'd get the best results if we don't think of it as a tool, but instead give it a name and think of it as our brilliant coworker!
Why should I trust you, internet rando over a stormtrooper-level salesman? /s
Brandolini's principle in action. It takes 10 times more energy to refute BS than to generate it. A related analogy to computing: it is easy to generate propositions, but hard to test if a given proposition is satisfiable or not, which curiously ties to P vs NP.
I much prefer the alternative name: the Bullshit Asymmetry Principle.
Watching for unexpected failure modes is surely worth it.
Intelligent life-forms can generate probabilistic outputs based on inputs, but being able to generate probabilistic outputs based on inputs is not what makes us intelligent.
Likewise, there is no reason to think the brain employs super-Turing or quantum computations that cannot be approximated by LLMs.
> exclude LLMs with CoT from the category of intelligent systems with certainty
At least, don't you think that the recent mathematical results of LLMs are a bit like a glimpse of something teapot-shaped in the orbit? (which makes it not a Russell's teapot, which. by definition, can't be observed).
To me, it's an expected progression of ANNs' approximation of human cognitive processes. The universal approximation theorem guaranties the existence of such ANNs barring the super-Turing or quantum superiority of the brain.
I don't? You are presenting opinions as if they are mine, but they are not.
They could have been great, if trained on datasets from a more sensible species.
???
Of course it is. The brain is mechanically not capable of doing anything other than that.
Do you believe the brain is something other than a bundle of probabilistic physical interactions? Or are brains not the source of what we call intelligence?
This is going to elevate your thinking on this no end, if you're interested.
We know that the brain is a probabilistic input → output machine because the universe is a probabilistic input → output machine. The brain is made of universe. There are deterministic relationships (which at high sensitivity or complexity become easier to describe as probabilistic), and quantum relationships. That's it. The brain, like every other thing comprised of "universe" is comprised of those two types of relationships.
If Romain's book provides evidence of relationships in the brain that are neither quantum (therefore random) NOR classical (therefore deterministic), then 1) he would have already won at least one Nobel prize, and 2) anyone in this thread would be able to at least gesture toward what relationship that is.
No quantum bullshittery in there I promise.
Equating "classical" with "deterministic" is however pushing it a bit too far, when no one and nothing can ever do even a very small fraction of said determination...
That's why the brain cannot possibly be anything other than an input → output machine, which is functionally deterministic (with maybe some fully random components), but is easiest to describe as probabilistic.
In the same way that LLMs are functionally deterministic, but easiest to describe as probabilistic.
The brain is an object in the universe.
The universe has quantum behaviors (fully random, not a source of intelligence) and it has deterministic behaviors (fully non-random). Many of those deterministic behaviors are so complex that they're easier to analyze and describe as probabilistic, which is where most brain input → output relationships land.
Please point to any evidence whatsoever that the brain has some third type of interaction going on that has never been observed anywhere in the entire universe, then we can have a discussion about it.
The brain is a (very complex, incredible) input → output machine. That's it! It's incredible!
I don't understand why people are so afraid of this that they will believe otherwise with literally zero evidence whatsoever.
The brain is deterministic at the level of specific interactions, which process inputs in a highly chaotic (but still deterministic) manner into probabilistic outcomes.
The opposite of deterministic is random, i.e. in the quantum sense of truly no relationship between input and output.
There are probably some quantum effects in the brain here and there, but the vast majority of it is just traditional deterministic interactions networked together in such a complex system that the resulting behavior is much, much easier to predict in probabilistic terms than otherwise.
You cannot say an AI model cannot be intelligent because it's a probability machine, when all available evidence points toward natural intelligence also being generated by probability machines (much more complex ones, called brains).
So, LLM are just next token predictors, brains are next token predictors + many other things in addition, and that makes people still feel LLM are dumb even when they solve a lot of problems using tokens.
Of the vast uncertainties and philosophical exercises that we must face to bridge the chasm between where we are now, and where we will be when we understand intelligence, I can take comfort in claiming, with 100% accuracy, that our biology is not based on technology invented by Google in 2017.
Like what?
What specific biological structure in the brain could be doing anything other than producing output as a function of 1) current electrical/chemical/thermal inputs and 2) previous electrical/chemical/thermal inputs?
The way they make LLM solve problems is by adding a lot of logical jumps into its data, or break down different problems etc, and then as it predicts the text it predicts these logical jumps and then solves the problem. That is very different from how humans learn to solve problems, you don't feed them a billion different state transitions they have to encode to be able to navigate math, they learn to become proficient at math from a few hundred to a few thousand examples, that is fundamentally different from how LLM can learn.
That LLM are so slow learners that requires massive amount of data is a big reason its hard to make them smarter, and its caused by them being next token predictors. And the reason humans can learn with so little data is because we are not just next token predictors.
You changed the definition there, for it to be like an LLM it should be:
> transforming an input into an output trying to mimic inputs that part of the brain has previously been exposed to
Anyone can see how that limits you a lot, and why that makes it so much harder for LLM to learn things properly than it is for humans.
Pre-training is just direct mimicry. A pre-trained LLM is very stupid and mostly useless. To become useful they are post-trained with a reward function.
On functional grounds my bike has not a lot of distinction from a horse, but just, like, saying that doesn't tell me much about either. Or at least, it seems to leave out a lot of otherwise crucial details and differences..
What does it mean to you, this point of view? Are you truly coming from like a 20th century pragmatism point of view? Where what is most useful is what is right? Or are your trying to make a larger claim about nature? I think being clear about that would help focus your critique here.
Are newly born babies reacting due to statistical probabilities that they have derived, or are they using something other than their brains?
The answer is obviously yes lol.
The creature is an assemblage of electrical, chemical, and kinetic relationships.
Watching a baby develop is exactly what you'd expect from a system that's predominantly electrical noise triggering behaviors and then gradually refining denoising the relationship between inputs and outputs, with the goal function of achieving more desirable inputs.
Surely you can at least gesture toward one thing in the brain that appears not to be a probabilistic relationship between input and output?
“LLM has made legitimate mathematical discoveries” —> Wow the rate of progress is amazing. Highly upvoted.
“LLM does something not good” -> Does everyone else not realize LLMs are just dumb next token predictors? Highly upvoted.
So tired of this discourse and this site.
These models can do a lot of things but they also can't do a lot of things. In order to use these models effectively you have to understand that they are next token predictors and how that allows it to do what they do.
For them to change the world you have to work with them as next token predictors. Ensure that the next token predictor has enough prediction paths to solve the problems you want and so on. Since when they don't they fail spectacularly. These big companies will continue to add new skills to them, so they will continue to get more useful.
You don't see how that is very different? For an LLM to be as smart as a human it has to be able to learn like a human. Like you don't evaluate how smart a human is based on how much he knows, you evaluate it based on how fast he learns. And LLM are so bad at learning its ridiculous, they lack that part of the brain that lets humans be smart and learn so fast and easily.
Yes, but not because they are useful.
The issue is of course with using the word "dumb": they are next token predictors, no doubt about it, but whether LLms as a class of system are smart or dumb is entirely unknown and entirely variable in time.
To interact with them effectively you must know how they behave, just like you have to know how humans behave to interact with them effectively. If you disagree, find someone with autism and have a conversation with them.
So people call them dumb since like dumb people they make strong statements about things they don't understand. And it doesn't matter how much smart things you encode them with, they will keep making strong statements about things they don't understand until they are fundamentally changed.
But since LLM are very smart about things where they have extensive data they can still be used to reliable solve many problems and probably in the future where we understand that better almost completely replace most lawyer and doctors work etc, because a lot of what a frontline doctor or basis lawyer work is very repetitive and can be encoded with billions of examples and decision paths into an expert system framework the LLM will follow.
So people say LLM are dumb since LLM will always keep making dumb statements. This is the same way we call Elon Musk dumb for making a lot of dumb statements, he is a smart guy but he makes dumb statements so her is dumb.
If ever there was a human quality.
Also, your explanation of "dumb" is really favoring the anti-llm side, and its a very generous interpretation. I suspect what is much more likely meant, is that token predictors cannot be smart, not now nor in the future after improvements, because they are token predictors and predicting tokens is not how intelligence works.
All of this is of course unfounded, and hidden behind the word "dumb".
Why do you think that? LLM are used as expert systems today, in order to quickly navigate problems by breaking them down and iterating between different well known possible solutions and paths to check etc. That is how they work, they do that by using their next token predictions, and for things they aren't well trained on they will produce dumb results.
LLM has solved enough problems that almost nobody has the view you ridicule here, but there are still many who think LLM are thinking just like humans and that you can trust them just like humans. So its important to remind people these are just token predictors and lack many things humans do.
> If ever there was a human quality.
Humans can avoid doing that by using introspection, LLM can't. That some humans do it by not using introspection doesn't mean humans are incapable of it, we know humans are capable of it, which is why we can point out when the LLM is wrong with certainty, humans as a group make extremely good predictions.
And thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.
This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.
This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.
Or should the discourse in a diverse community like HN only reflect the positions you personally hold?
My point is it's silly to whine that HN is a place where multiple points of view on the topic are aired out and discussed.
If you want a personal echo chamber where only your own beliefs are affirmed and anything else is flagged off or downvoted, I'm sure you can go find one or, worst case, vibe code one into existence.
I get the impression you want me to concede that the particular points of view you disagree with aren't worthy of representation here on HN.
I'm not going to do that.
Since we disagree on the present let’s informally do a “remind me 2 years” to this discussion and see what’s happened then.
You mean your particular version of it.
It's interesting to see you consistently missing this point.
You've decided LLMs are clearly more than just complex but mindless statistical models.
You've decided that based on, it seems, the very impressive things these tools are capable of.
Therefore if anyone claims they're just mindless stastical models--with or without any attached judgement as to their actual utility or usefulness--then they are ipso facto wrong.
(And yes I just used endashes, damnit!)
That's on you.
It is in fact possible to simultaneously believe that LLMs are mindless token predictors and that they're enormously powerful.
These are entirely orthogonal beliefs.
Heck you could equally believe that LLMs represent true emerging AGI and that they still remain deeply flawed and are only an incremental step along the path of automation.
Or somewhere in between.
And discussing that space of possibilities is, I'd hope, precisely what HN is for.
"These models are probabilistic, you shouldn't blindly trust them in spaces where accuracy is really important" seems like pretty sound advice to me.
I mean even perennially contentious topics will get this behavior.... some thing about emacs makes the front page, within a day or two there will be a vim post up there. Same with Rust is (good|bad), or if systemd creates an even more awesome tool, the haters will come along and recycle stories about bugs from over a decade ago.
There's a lot of people here. Not all of them read it every hour, and discussions like this among large groups often take a very long time with lots of repetition. Human group dynamics (aka politics) is slow.
> So tired of this discourse and this site.
You're welcome to leave if you don't like it. The site was like this long before you joined, and will like it long after you leave I'm sure.
It's also worth noting, that an awful lot of math discoveries are perfectly in line with dumb next token generators - they are finding a way to formally construct an argument and being surprised when it doesn't work, or surprised at the outcome of the grind. Not all of them are made by brilliant leaps of intuition.
I agree that humans must verify LLM-produced facts, but strongly disagree with these kinds of "stochastic parrot therefore dumb" arguments.
Yes, an LLM is a "stochastic parrot". No, that doesn't imply that it is dumb. Enough to look at how Terence Tao asks ChatGPT to help him understand a solution that nobody had ever discussed before [1], or how a random guy asks ChatGPT in a handful of words to disprove a 30-year-old conjecture, with zero technical input [2].
If your parrot in a birdcage with internet access can finish the sentence, "The counterexample to the Dinitz–Garg–Goemans conjecture is...", then it's a pretty smart parrot, by all reasonable definitions of "smart". Just because someone bottled up the formula into matrix multiplications and added some random sampling to the outcome, that doesn't take away from the fact that the parrot said provably correct statements that the biggest experts in the field couldn't imagine.
And no, I'm not implying that the LLMs are correct all the time, or that their intelligence and reasoning works in any way like ours.
[1]: https://chatgpt.com/share/6a5fdc7a-d6f8-83e8-bbea-8deb42cfed... [2]: https://chatgpt.com/share/6a60b2eb-0b64-83ee-9c76-7931ca1de0...
But the LLM is still dumb where those skills doesn't have good coverage, since unlike the expert systems it maps fuzzily to its skills, and they are tuned to produce results over rejecting the request when its unclear if coverage is good. As long as that is true you have to treat them as dumb even if they sometimes produce brilliant results.
This line of critique is pernicious because it is both technically correct, as description, and profoundly misleading.
Saying that outputs are a product of inputs is not interesting and to the point it is not explanatory.
What is interesting, is how they do what they do. What is the "statistically likely* next token? To answer that you can do exactly one thing, run the LLM. That's because what they are doing is interesting and not reducible.
What is more interesting is that in order to do what they do, given the architectures we apply and the training strategies we use and the harnesses we situate them in, LLM are recapitulating in their deep layers strategies observed in the animal brain. This is still suggestive, interpretibility is nascent: but it is also more than a little interesting. In some respects, for cognitive scientists interested in the manner in which mind merges from computational substrates, it is profoundly interesting.
One can incorporate this, and, still be viciously critical of bother the success and failure of LLM in the applications we have put them to, and of how we (as individuals and as institutions such as corporations) are integrating them into our work.
There is a lot to criticize!
But criticism can be taken more seriously when it is not obscured by misunderstanding or misrepresentation (intentional, or not) of what LLM are and why they are not remotely "parrots" in the pejorative sense.
The technology, as technology, at the scale we are architecting it, is doing things we did not imagine would be witnessed in our lifetime, if ever. Dismissing that and denying it because of the career, industry, society, and civilization challenges that technology brings are existential, is bad argumentation or bad faith.
Both can be true at once.
I see no credible corroboration. More likely its folks having no more care for what they are doing than the bots themselves.
> Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this usage of “AI”.
People said the same about email spam ... until they engaged spam filters. CVE report slop is simply spam. Complaints are better directed at the filters, not the filtered.
I didn't get this at all from the parent. They're simply stating that LLMs aren't entirely trustworthy, and that the responsibility is ultimately ours, not the LLM's.
Each layer of attention can more through feature space “lit up weight clusters” in a way no other previous AI can. It can from that decode some rudimentary logic and world modeling and make deductions. Certainly better than any previous AI. Only a goof here would believe this wasn’t a serious advancement.
So don’t over sell it. But don’t sell it short with this “grrr in an engineer don’t threaten me with new tech” attitude.
This take is akin to teenage angsty takes and doesn’t really belong here.
I hate AI slop as much as the next guy but the amount of tribalism over AI is taking near-religious forms.
Nobody knows what intelligence is, therefore we don't know what does or does not possess it, therefore we don't know whether LLMs currently, or in the future, possess it.
Yes, LLMs can be stupid, guess what: so can I. That doesn't really change the argument at all.
I feel like I'm on a deja-vu from when DALL-E was released and everybody was fighting over whether AI can be creative yes or no. Same story, different words.
Intelligence, creativity: we have no idea what these words mean, and AI is helping us understand them better. That alone is an achievement of epic proportions. I am not joking here. Any computer scientist before 2015 would be absolutely blown away by what you can now do for 10 cents and an API call, yet somehow because of the tech-bro-iness of it all we get a tribal war over what is plainly visible in front of us:
LLMs are uncomfortably close to what we thought intelligent machines would look like
Some predictions might be the tokens "I don't know", but that is based on the model mapping your text to those tokens by having seen many similar "I don't know" responses to such contexts, it didn't do any introspective logic to produce that "I don't know", and its possible it actually does know if it followed another branch there so "I don't know" is often not even true.
If they had an introspective part that stops the prediction when its too unreliable it would no longer just be token prediction engine, and I believe we need such a part for them to become what I call smart. I don't think LLM will ever stop being dumb without such an introspective part to them.
And no, that introspective part is not a part of the token predictor. At least not in us humans, the feeling of certainty we have is not a prediction, it is bundled with our thoughts, so we get both "answer is a bear" and "certainty is low", we don't get just one of those as a "prediction".
Will LLM become smart as humans with such an introspective part? I don't know, but I think they will never become as smart as humans without one.
Note: The certainty score has to be per conclusion or response, not per token. You can't evaluate a responses validity by aggregating the weight of each token. Meaning its a logic engine, not token engine, that evaluates the certainty of a statement being correct or not instead of a token being correct or not. That is the level human thinking works at and seems to be dramatically more efficient.
Maybe our ability to reason is not as mystical and special as our ego might hope it to be, and discomfort over LLM's bringing that to light is the root of some people's urge to continuously downplay and discredit them.
(As far as I can tell as a non-neuroscientist, the literature on how our default mode network and prefrontal cortex interact agrees with my assessment - free association and logical verification respectively)
That introspection isn't an illusion, what your consciousness see of your thoughts obviously are tings that has been calculated and aggregated by the brain, so we know the brain calculates and aggregates those thoughts and feelings to produce its results. And we know LLM doesn't do that, it doesn't have a side system that does that sort of introspection.
But it does that introspection, we evolved to make it. If its not useful for anything we wouldn't have evolved it, it can't have been easy to evolve a consciousness so there has to be purpose for it.
Or do you think our consciousness is a magical ghost thing unrelated to the brains workings? I think its pretty obvious that our smarts in part depend on the computations that results in our conscious experience, you need a very strong argument as to why that wouldn't be the case. And broken brains being broken doesn't answer this.
The very second they add this part, it will "just be a dumb token predictor with introspection", mark my words.
AGI is fairly easy to detect for this reason. Does this system make a majority feel you don't have to hire people anymore? If not its not AGI.
Solves what? Chess? No thats not AI, its just a chess bot. Turing test? No, thats not AI, its just a dumb token predictor.
Why would the goalposts stop moving at AGI? I am 100% convinced it will somehow still lack the "gusto" or the "taste" or the "timbre" of real intelligence. You can see it in coding right now, AI has bad "taste" in coding, because really, we can't do a better job critiquing that which is obviously (on occasion) just plain good.
You are moving the goalpost here if you think the chess AI was AGI. All those problems were evidence AI wasn't as smart as humans, not goalposts determining when AI is as smart as humans. The first turing test winner wasn't even an LLM, it was an expert system, so we already knew that test wasn't enough for AGI.
If I want my ox-cart to fly, I need to add wings, but that's probably not all that needs changing.
I've thought about this for quite some time now.
No. A human doesn't need to verify everything. And the argument is really simple: stochastic.
Think of self-driving cars: We can show today - based on evidence and real data - that self-driving cars are safer than human drivers. That's a fact and the consequences are clear, more self-driving cars, less human-driven cars, less accidents, less hurt people, less dead people.
Are the cars 100% safe and NEVER make a mistake? No. But they don't need to. Nothing is ever 100% (in the real world).
Now back to AI for software creation. "Review is the bottleneck because EVERYTHING must be judged by a human." No. It doesn't. We just need to build AI review systems, that will do reviews better than (or at least as good as) humans. The human review quality bar is far below 100%. Far far far. If we can show (likely in the next 12-24 months I think) that AI review quality is consistently above the human review quality - again, based on evidence, based on real data - then that's it, then there's no good reason to have humans review the code.
Yes, there will be another layer in the system, another level of abstraction that will/must end at the human boundary.
Let me know once the majority of software engineering organizations start only checking in markdown files and let code be generated non-deterministically from these specs in CI. If this is not happening now, there’s clearly a sufficiently high level of distrust in blind LLM output (both code itself and reviews).
And even then, are you suggesting humans shouldn’t at least review the markdown specs? Why not have LLMs review the specs then? Is there, perhaps, some fundamental quality to human review process that is desirable here?
We have plenty of systems where complete accuracy is the only acceptable thing. Computers are great for such things. Until we all get caught up in a way of delusion and start writing those systems as natural prose passed through an improperly understood stochastic machine.
Flesh-based “brain” is able to use its vast corpus of inputs and calculate the most statistically likely output in a given situation. It is probabilistic, and when you are dealing with probabilities in a situation where certainties, not probabilities, matter, you’re going to get dinged on credibility massively when your flesh-based brain gets the probabilities wrong at best, or in this case, claims a line of code generates a vulnerability when it is, in fact, a code comment.
Humans are prediction engines. They are not Pure Intelligence, and shouldn’t not be treated in any form or fashion as if they possess pure intelligence. What bothers me about this entire situation is that presumably the folks that have relied on the flesh-based “brains” to generate these vulnerabilities knew (or should have known) enough about their "tool" to know this would happen, but did not: To err is to be human.
Now, we all pay the consequence, to the tune of hundreds of thousands if not millions of dollars of wasted productivity from teams that have to deal with the resulting fall-out of this over reliance on fallible “brains".
A human must verify everything another human presents as fact. Everything. If you don’t, we all pay the price. Using a human does not remove the onus of responsibility on the human being in charge, if anything they amplify it because humans work for peanuts in some countries, and can generate lots more output more quickly that needs to be verified by the humans in charge.
However, error margins are in the center of any engineering discipline. We cannot produce things measured with 100% accuracy. This is accepted fact. The focus is always not on eliminating errors, but on reducing them to acceptable minimum. With LLMs we should not expect an ideal logical thinker, but a process that may error sometimes, and we must design quality controls instead that push LLM outputs within acceptable margins. And it can work.
In the current AI mania, there's a lot of due diligence simply being ignored. Plenty of "Well humans make mistakes too!" going on here on HN too.
Risks are not binary. Most people are not idiots. They do understand that LLMs aren‘t thinking humanly or 100% logically. You are talking about some sort of faith, I think this is more about trust, which is built on observations. Statistically, the outcomes of LLM work may hit the goal quite a lot. More of that in the beginning of the journey than in the end. For many reasonable people that becomes a trap, where minor, acceptable deviations accumulate into a fireball. Still, saying that LLMs are failure with inevitable fireball at the end and giving up is a trivial and stupid solution. Staying within razor-thin distance from that fireball and managing to deliver a working solution is what everyone tries at the moment. This is how it always happened and always worked. It‘s the art of engineering.