I think a lot of model architecture changes don't move the needle ~that much and are basically done to improve runtime performance and scaling.
- There's tons of cases of ml researchers, tech companies, and labs that propose new and improved models that turn out to not really be significant quality improvements (if at all) once the community starts experimenting with them.
it does seem like (IMO) the biggest leaps have basically been perf & scaling + RL on high quality data + harnesses.