But in the case of CPU architecture there are SOME people who understand how things work 100%, and they've built and vetted abstractions/mental models that enable other engineers and scientists to have that kind of mixed shallow/deep understanding in a way that works. On the side of LLMs we're still lacking an expertise which could flawlessly explain how these things operate; the abstractions that we're using are instead derived inductively and are totally unvetted.
replyYou are right that the field doesn’t have a theoretically sound explanation for the architectural choices aside from “A works better than B”. However, I would argue this is an ideal opportunity for the “gentleman scientist” or eager 17 year old.
replyBasically every part of the original transformer was replaced with something more efficient or better:
LayerNorm -> RMSNorm
Sinusoidal position encoding -> RoPE
MHA -> GQA
ReLU -> GELU
What this means is that there is ample opportunity to improve on what we’ve done thus far.
Oh yeah I absolutely agree that there's lots of greenfield for relative newcomers to find some big improvements! I'm just saying that reasoning about 'best practices' in terms of what works for something like a CPU is missing some important and relevant differences.
replyIn fact, one of the jobs of an engineer is to make sure that other engineers who don't work in his or her area do not need to understand that area deeply, yet build something reliable with it. They need just the summary that he or she writes up into the datasheet for the part. Ensure these conditions are met for safe/reliable operation, give it these inputs, expect these outputs, these timings, this energy consumption, this heat generation, frequency response, tensile strength, whatever.
reply