Yes, all three together would be even better. But it wouldn’t be if you had 100x more fanfiction, mostly synthetic, generated during RL to teach a model to be better at writing fan fiction. There are real limits to the amount of knowledge you can cram into fixed-size (downstream of hardware availability) weights. For a period scaling with data was basically “free” because we had the Internet and all the books/media that humans had already created; the data was accessible and limited (at least, the parts we think models should know about) enough and top-hardware big enough that we could basically compress the whole thing.
Post-training/RL are making this obsolete because they’re more about skill/capability acquisition rather than knowledge. They can generate much more data (most of it quotidian/useless, ie an agent made a typo in batch 382829) and clearly seem to cause a kind of mode collapse even in the most advanced frontier models.
We don’t need to make LLMs forget about SpongeBob SquarePants so they learn more about bash. But if I have a question about SpongeBob SquarePants, I don’t need to hear about load bearing seams prefaced with honest caveats after a model writes 400 lines of bash to look up SpongeBob’s family.
And there is probably a lot more SpongeBob knowledge we could put into models if we wanted to: interviews with the creative staff, a SpongeEnv/SpongeHarness modeling how the art/story team work together to create entertaining kids tv, a SpongeBench measuring entertainment value, etc. If a SpongeAgent spends 2000 years in Agent University learning how to Spongemaxx we probably don’t need or want to have it spend another 2000 years writing smoke tests
You just defined a liberal arts education.
- Understanding of protocols like HTTP.
- HTML, JS, CSS, SVG, and everything "web".
- Understanding of databases, SQL, etc.
- Abstract code architecture patterns.
- Understanding the users' requests in English.
- Responding in English.
- Command line tool usage (agents/harnesses)
- Industry-specific knowledge that can be applied.
- Frameworks, SDKs, applicable libraries.
- Relevant legal requirements.
- Etc...
It would be trivial to have a pre-flight convo with an llm to guide the user thru module choices. "Build a site" -> "ok, describe the purpose" -> "local council in XYZ location" -> "that implies you won't need localization since XYZ has a monolingual government" -> "english and catalan localization please".
Right now, you prompt and it builds using assumptions, and we prompt to adjust. I think it would be great to be able to pre-load a set of assumptions.
Everyone assumes that carefully crafting a specific AI architecture with bits and pieces bolted together based on their human intuition is necessarily superior to simply using a bigger monolithic AI model. It turns out that the opposite is true, and has been demonstrated over and over again.
The bitter lesson is this: You can simply ask a frontier model to do the things you suggested, in a few terse lines of English. Dump a few lines in AGENTS.md and you are good to go.
Your approach is to "fiddle with inadequate tools" for weeks or months until you can finally attain a pale imitation of what the frontier models can do effortlessly.
It's the classic "But I can customise EMACS endlessly, why would I use an actual IDE?" argument all over.
I get it. You don't feel ownership over someone else's AI. You don't feel involved, you don't feel like you have agency.
It's like LEGO or IKEA furniture: study after study has shown that people enjoy things more if they "put it together themselves", even if fundamentally the thing is worse and/or still essentially nothing more than plastic made in a factory.
There’s no need to bring religion into this.
You don't _have_ ownership of someone else's ai, and that comes with real risks.
Security risks, privacy risks, business risk.
They might rug pull you, they might charge you more, or like atrophic, silently corrupt the answers, or code...
The labs are happy to jump on any emergent capability the scaling and training impart: generate prose, teach you things, cyber security, design, code, etc.
Do you really think that the frontier labs won't turn a popular capability, or trend they notice, into a first party tool if the ROI seems there? If it's your own private ai in your datacenter, you can keep it all secret, and not lose your business.
On the bitter lesson you're right of course:), wish I had a super computer to just scale that instead.
A) You can always self-host something like Kimi, DeepSeek, or GLM.
B) Just because you use a specific proprietary AI for programming doesn't actually bind you to that provider in any meaningful way. The authored code remains even if you stop paying them!
Of course, if you use AI as an active component in some sort of service, then the EULA, rug-pulls, etc... suddenly start to matter. That's a different story.
That paragraph sets me off. I’ll take Vim and Emacs over VSCode and Eclipse any day.
And of course the neural network series.