If I understand correctly the basic structure hasn't changed since Transformers were invented. It's just gotten bigger and the models have had better training and also some optimizations. I don't think this one has optimizations.

No way! It is a while since I have seen that extension here on HN, I couldn't be mistaken. With AI? Is Pascal still alive?

the math involved is not very hard to understand. it’s linear algebra. the transformer model is brilliant but simple, nobody even really realized the impact it would have until they started training it on massive datasets