Same issue happens in models trained by organizations who aren’t selling tokens. I believe it’s because being parsimonious is simply harder. Achieving the task at hand independently and declaring the job done is easier than building an abstraction and reconciling between every use case.
replyLabs are trying to make long-horizon work. Even if you're a coding agent, adding more and more surface area is distracting to that goal. There is reason that RL over long traces should, at least in principle, optimize for building in ways that help the result fit in the model's context window.
replyA meaningful risk of course is that the tools available to the model (ripgrep + fancier semantic approaches) allow it to do a good job of reasoning over things much larger than its context window, and so it doesn't pay the penalty sufficiently to fix it.
Does that not sound a little silly to you when you say it? Should I invest in becoming a memory athlete as a way of becoming a better software engineer? ...or should I learn how to build and use tools?
replyWhile I don't disagree, memory certainly was more of a restrictions on us humans than it is on llms. Therefore, the answer may not be as obvious as it seems. We build abstractions to reduce (memory) footprint of features, right?
replyAll the open weight models, that are given away for free, across orgs and even nations, are using the same methods to achieve high performance.
replyWhat's more profitable, optimizing for inference time or optimizing to increase inference time by increasing token count?