> This is interesting to me because reducing context & token usage is in the user's best interest but not in the financial interest of AI vendors.
replyAI vendors still need to compete with each other both in terms of token cost and competency. An agent that is costly and less effective by wasting tokens is less competitive.
The tokens are still being burnt, they're just doing so in a parallel dimension from the users main context window.
replyIt's true that the initial tool response still has the same amount of tokens but it doesn't keep dragged along in the longer-lived top context.
replyThe real benefit is being able to use a cheaper, but good enough, model with a specific system prompt dedicated to that task.
reply