upvote
But to extender your analogy: We do understand how Claude does what it does and how it is built, so how is this different?
reply
I would dispute that we completely understand how LLMs do what they do. An analogy would be that we meet an alien race that gives us their proof of the Riemann hypothesis translated to Lean. Plus of all other provable mathematical conjectures we come up with. And we might kind-of understand how those aliens do math. It would still be a bit sad that we'd be relegated to consuming what the aliens give us, and never be able to ascend to the level of the alien mathematicians
reply
We didn't meet a hypothetical alien race though. We created it. We are giving birth to something that will soon surpass us in every conceivable way.
reply
That doesn’t make much of a difference. It’s alien, and for the most part we didn’t design it to do the things it turned out to be able to do by happenstance. We might as well have grown the aliens in a test tube by some unexpected gene manipulation.
reply
Over half the cells in our bodies are not part of us. The lines we draw are seemingly never respected by nature.
reply
An analogy here is evolution. We understand what it does on a basic level. There is natural selection and species that are well adapted to their environment thrive.

This absolutely wouldn't allow us to predict what species will emerge and that something like humans would be the dominant species.

Similarly, we understand the mechanics of how LLMs work, but that does not make it predictible in any way.

reply
We definitely don't understand how Claude does what it does! We know how to grow a Claude, but we don't understand what goes on inside a Claude.
reply
I remember when about 15 years ago, I wanted to make my neural network home work based on proven math. I had to train a model to learn basic number recognition. The math said 100000 neuron is what's needed, so I typed in that, and my laptop immediately froze completely, and died a little later. I learned on that day that we know empirically that a few 10s are enough.

We have the same thing with LLMs. We don't know how they work. We don't understand them. We have only empirical, statistical indications, nothing is proven. We know how the small parts behave, we really understand those, and also the different algorithms. But they whys? No. We're getting better, but still far away.

reply
If we don't know how LLMs work, how did we create them? They didn't just spawn suddenly on someone's hard drive.
reply
What does 'know' mean? And what does 'mean' mean?
reply
Trial and error. A ton of trial and error, and a lot of guesses. More wrong than good guesses.

Basically every science field is based on this.

reply
And presumably mathematicians understand the output, the proof, whatever it is that came out of the process. I don't feel like humans have to be the one doing all the manual math and figuring out all combinations.
reply
I mean I understand how cars work but that doesn't mean I'd feel the same accomplishment of driving 42 km as running a marathon.
reply
We already had AlphaFold, which "solved" the protein folding problem, but scientists readily admit that they know next to nothing new about the field, despite it being accomplished.
reply
I mean computers made Apollo work. They couldn't get it done with human paper calculation. The lander itself had a processor in it. So its kinda the same thing. "Humans should have done this by hand," is just an odd thing to want and ties back to some really backwards thinking. These are our tools. No one is expecting us to make homes without power tools or walk for miles instead of taking a train. At a certain point AI will be normalized and we'll have actual critical takes on AI that make sense other than "Well, I just don't like it for vague human achievement reasons." The same way people complaining about 1980s cars getting "computer chips" and "no longer being able to hand tune my carbonator" have long accepted more complex and automated and computerized cars.

As far as not knowing how it works, that becomes a bit of a philosophical problem. We know how LLMs work and their algorithms. I think we can always play up "we don't know" is disingenuous ways. How many people who made Apollo calculations could also write long proofs and theorems of the fundamental maths they were using from first principles? Not a lot. We're always standing on someone else's shoulders. We'll never truly know everything or how exactly our tools work on an individual level. Maybe AI changes things because an innovation may not have a direct line to an Isaac Newton at its base, but I think that's a bit of a vague and poetic difference, and not a real one. Its still human achievement but via machinery.

I dunno, its a bit scary, but I do feel like its just continuing the same line of human ingenuity. Not knowing exactly how something came about and non-determinism scares some people, and perhaps rightly so, but pretty much all major technological advances are scary on some level because they so strongly challenge the status quo and our ways of thinking and or ideas of what is good and right and proper. At a certain point, its hard not to see the more dramatic AI outrage as just hysteria and moral panic. I find it interesting how easily we can spot moral panics in the past in history, but when its our turn, it can be very hard to see when we're in a moral panic. To many living during those times, the moral panic sounds entirely reasonable, but its still a moral panic.

reply
people move goal posts
reply
And if the rocket failed at any point nobody would know how to fix it. Only the AI guessing and saying "you're right to point that out" when the rocket explodes and everyone inside perishes.
reply