I think what you’re seeing here is a difference in the amount of “world knowledge “ encoded in the perceptron parts of the model as opposed to how good the model is at the “transformer” part which you could think of as pure token prediction using only what’s in the context window.

If true that would suggest gemini/gemma would be great in a RAG situation where world model isn’t needed as it’s being spoonfed all the relevant information and less good at green field tasks.

That’s interesting to me because I have been struggling to understand how gemma4 is so good in my local use and how notebookLM does such a great job does when I give it project docs and yet gemini has always seemed behind claude when I use it cold for stuff.