This is a great point. The complexity sources are entirely different. An OS is complex because it’s solving a difficult problem and we’ve seen decades of evolution both in knowledge and-how and, at the frontier, in performance standards.
replyThe abilities of LLMs are emergent. You can experiment with LLMs and know as much about their observable behavior as the experts. But there’s no way to “crack open” an LLM and see precisely where each skill or tendency lives; as far as we currently understand, it’s all mashed together.
Learning how to build emergent systems is also a skill kids should learn these days. The closest I got was coding up game of life for CSE 142 (intro programming).
replyIf stochastic gradient descent isn't taught in whatever CS Theory 101 is now, it really should be these days.
Yah, "building" it is not sufficient. But a lot of times when I build I want to know the why. "Why does gradient Descent have some clever tricks that easily translate to matrix math"?
replyLot's of neat stuff to learn.