upvote
Synthetic data allows them to train well past the limits of human writing.
reply
What's an example of synthetic data?
reply
Only in the same sense it's not yet determined about humans, either.
reply
Not so sure. Was everything already "there" before humans existed?
reply
In some form and shape, yes. Humanity's creativity is a lot of marginal copying and remixing.

But obviously, it adds up to something greater than went in; in aggregate, our contributions are something to awe.

But my point is, if you zoom in at the marginal, incremental contributions of any individual human in this process, it's really hard for me to say LLMs are not at the same level already.

On this topic, people like to compare LLMs to Einstein, but as far as I know, Einstein did not zero-shot special relativity in an afternoon. He built it up incrementally over time, it took him three times longer than the time between first ChatGPT release and today, and it depended on centuries of prior art, culminating in the right observation and right notation being available to him in his moment of greatness.

reply
Unless everything was there before humans existed humans created some ideas etc from scratch and not just remixed and copied.

At what level LLMs are is then an entirely separate discussion, I think.

reply
> humans created some ideas etc from scratch and not just remixed and copied.

Name three.

reply
What would you accept as evidence there? Are, for example, the first names/words for colors from scratch?

So your view is that everything was there at the creation of the universe (it's a possible view, of course)? Or are there any "things" that can create ideas from scratch?

reply
Recently I watched a documentary on the tanzanian Hadza tribe, one of the last hunter gatherer tribes on Earth. Their language is a distinct click and pop language and they regularly imitate animal calls (monkeys, baboons, birds) when they hunt but also when they communicate with each other, tell stories etc.

I think it's not impossible that words evolved as adaptations of the environmental sounds with which our ancestors lived. The human creativity producing DNA is also a remix of preexisting molecules formed under evolutionary pressure, so the view that it's turtles all the way down, unintuitive as it is, may not be so indefensible after all.

reply
I mean what is your criterion on invention here? On the one hand, each specific word could be seen as a new invention. On the other hand, all languages basically correlate strongly with the environment of their users - it's why LLMs turn out to be universal translators - and pattern-matching is hardly an invention, isn't it?
reply
But LLMs aren't turtles all the way down, they stop at vector embedded tokenized words.
reply
My view is that LLMs meet the standards by which we judge human creativity/inventiveness, and thus that one cannot claim LLMs "just repeat, never invent" without the same being true about humans.
reply
Pornography, "I Want it That Way" by the Backstreet Boys, torque wrench.
reply
No this is not an issue. As long as their is a way of verifying things, they do the same thing with creating novel things as humans: Searching through an infinite space of possibilities opitmized by knowledge.

They combine things, verify it and if it works and progresses the problem, they created something new.

reply