I’m fairly certain that in a few more releases we’ll have models with shorter CoT chains. Whether they’ll still let us see those is another question, as it seems like Anthropic wants to start hiding their CoT, potentially because it reveals some secret sauce.
The one which maximizes ROI will not be the one you rigged to cost more and take longer.
Directionally, tokens are not equivalent to "time spent processing your query", but rather a measure of effort/resource expended to process your query.
So a more germane analogy would be:
What if you set up a laundry which charges you based on the amount of laundry detergent used to clean your clothes?
Sounds fair.
But then, what if the top engineers at the laundry offered an "auto-dispenser" that uses extremely advanced algorithms to apply just the right optimal amount of detergent for each wash?
Sounds like value-added for the customer.
... but now you end up with a system where the laundry management team has strong incentives to influence how liberally the auto-dispenser will "spend" to give you "best results"