upvote
> What is preventing AI from continuing to improve until it is absolutely better than humans at any mental task?

No matter how much compute time it's given to combine training samples with each other and run through a validation engine it will still be missing some chunk of the "long tail". To make progress in the long tail it would need to have understanding, and not just a mimicry of understanding. Unless that happens they will always be dependent on the humans that they are mimicking in order to improve.

reply
What is the difference between what LLM's do and "true" understanding?

I feel like people grasping straws on the shrinking limitations of AI systems are just copying the "god of the gaps" fallacy

reply
> What is the difference between what LLM's do and "true" understanding?

The thing where you can understand the meaning of this sentence without first compiling a statistical representation of a 10 trillion line corpus of training data.

Unless you're an NPC of course.

reply
I mean brains get a lot of training data too in order to understand language. I don't think you provided a relevant difference.

Or rather, maybe I don't understand what you mean :)

reply
When you think about the word apple and what it signifies, what do you experience? Is there a feeling of "appleness"? Do you think that sense of meaning is equivalent to the numerical weights of an LLM?
reply
When you think about the word apple and what it signifies, what do you experience?

So I have all sorts of associations with "apple" and spent a little time playing with it.

First in a raw physical sense I can imagine an apple in my head, spin it around, imagine its physics with near cylindrical symmetry etc. A red apple is what first pops into my head, although of course I know there are many apple variants and have opinions on their taste etc.

There are many cultural associations I have with apples from Newton to George Washington. The company Apple has its own set of ideas that I interact with when I hear the word.

In other words I can think of various associations I have to the word apple of various strengths. These associations and strengths are functions of my experience encountering the word and actual apples.

Is there a feeling of "appleness"?

I don't really know what this would mean. I would say no, unless it can perhaps be defined what appleness means and feels like. I don't really notice any strong set of emotions or feelings from this thought exercise.

Do you think that sense of meaning is equivalent to the numerical weights of an LLM?

Again I think I would need a definition of "sense of meaning". I don't seem to derive a singular pointlike meaning when contemplating a singular word. I never was contending that human and LLM cognition are exactly equivalent, but I could see these association strengths being represented in LLM weights. I would say then if an LLM has similar association strengths with "apple" then it "understands" apples as well as I do. Of course this is really hard to test, but frontier models could give you all sorts of apple facts and cultural associations and so on. It may slip up and hallucinate, and I'm sure that I also believe at least one false thing about apples.

So what is your brightline between LLM and human understanding in this example? I assume that your line of reasoning would argue that LLMs do not understand apples. Why don't LLMs understand the word "apple?

reply
deleted
reply
That’s one possibility. If it fails to convince a critical mass that it’s a net improvement in their lives, then the impediment to continual improvement will be sabotage.
reply
I think there's been natural but steady progress with since 2024 with the release of the o1 model, which showed impressive reasoning capabilities. But I think it's wrong to look at the magnitude of the accomplishments and assume that will be field independent. We don't know the range of problems reasoning techniques are useful for. What we see here is refinement of capabilities that have been noticeable for years.
reply
> everything we care about

One qualitative distinction that remains for the time being is that humans care about things while AIs do not. Human drive and motivation is needed to have AI perform tasks.

Of course, this distinction isn’t set in stone.

reply
Maybe after decades. 2022 models were microscopic compared to latest models.
reply
It depends on if AI can invent cold fusion before running our of all the energy on Earth.
reply
You should really look up a video about what GPTs fundamentally are.
reply
You should also really look up a video about what neural synapses really are.
reply