I'm looking forward to seeing what types of new things people create over the coming years once there is less obsession with massive unwieldy LLMs. I think the incentives are just too strong to ignore.
With the benefit of LLMs already being proven, in a couple of years we will have vastly better hardware for inference I guess.
I feel like now hardware is stagnating a bit, because the software side has moved too fast for the hardware to catch up. Once we settle on some good, optimal software architecture for the models, dedicated hardware will easily increase throughout by 10x or 100x, for a fraction of the cost.
LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
Maybe we'll have model weights being shared like game cartridges.
Well, everything—even human life on Earth—is just temporary, but RAM supply lagging centralized-AI-driven demand increases continuing to squeeze the consumer market may not be a short term phenomenon.
> LLMs seems quite simple, maybe we'll be able to print/assemble at home our own chips with the desired models/weights.
So, the solution to the RAM crunch is “everyone has their own home chip fab and deals with the raw material supply and hazardous waste disposal”?
I...don't imagine so.
My idea was more like you get some pre-made chips, that you can maybe assemble together configure at home with your desired models.
Maybe each one of this chip is a layer, so you can stack as many layers as you want.
This has a lot to do with how MCTS works BTW. The current best path is often only the current best path because a lot of investment has been sunk into it. If you were to put equal resources into a different path you may find that it was actually far better. It is just that the early rollouts favored the other 'best path' so you sunk a lot of resources into that one. We are very early in our exploration of LLM architecture. I highly doubt we are anywhere near the best path right now.
The diffusion models are interesting, but those also seem hacky.
I think the next form of AIs will be simpler and more abstract.
The building blocks of our brain don't have the notion of a "token" embed into them, it's lower level that that.
I think first step is to find a better way to represent information.
LLMs shouldn't "compute" stuff using language tokens, but some other, more efficient logical mechanisms. LLMs should first "feel" the solution, reason internally in that optimised space, then, only when interacting with a human should it convert all that into actual tokens/language.
The other option is maybe hook up humans to EEG or the likes and map their brains while they solve different kinds of problem, or just see and feel the world around them
I'm having an extremely hard time thinking of companies that have prospered due to software optimization. Most of them were swept away by hardware advances, instead.
I think when successful, optimization really just means 'finally built right' and people forget the ridiculously inefficient ways before.
They weren’t quantitatively better than previous companies. They were qualitatively better.
I don't think people are realizing that speed can allow for categorically different user experiences that are more than just "worse than frontier capabilities but faster".
And I'd argue "hardware advances" are more proof of optimization.
This is 100% true for pretrains, likely true for RL as well although maybe there is some benefit to smaller activated params there. There is of course 0 benefit to small dense models relative to large sparse ones that are equally as memory efficient if you have enough computers.
Many on HN are in deep denial about this imo.
Smaller models are suitable for simple tasks like classification / summarisation while larger models are better in agentic capabilities.
Now, the labs spend enormous effort curating data pipelines to fit the models to a large assortment of very specific tools, tasks, harnesses, domains, etc. They also kind of fit to benchmarks by creating loads of synthetic training data that resembles benchmark tasks. None of this feels like the “scale up primitive methods and turn off your brain” message Sutton originally delivered.
While I do love optimized software, the hardware side, especially for PCs, has been stagnating for way too long. At least now we have a valid use case for doubling available RAM every 2-3 years again.
I had a reasonably beefy Lenovo consumer line laptop that I bought in 2011, 8GBs of RAM. Its screen hinge broke and I couldn't repair it but I'm fairly sure it was otherwise still usable in 2023-24, once the HDD was replaced with an SSD. I think even now entry level laptops are sold with 8GB of RAM.
By comparison a PC from 2000 was utterly unusable in 2012-13.
I got the feeling laptops gonna feel very different in 2036.