It’s kinda funny that your last paragraph is basically describing why sparse files, sparse matrices, etc. are used in other contexts.

It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.

Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…

We certainly would know where they were if they were just in smaller models in the first place!