IMO, the root of all of this is the almost total inability for most managers, and most eng orgs to measure individual engineer output in any useful way. And in particular in a way that lets you reliably compare engineers to each other.

Despite decades of the industry telling itself that we "pay for performance" or whatever, that has never been the case because we can't really measure performance very well. Where I have seen it done ok (not great, just ok), it was massively labor intensive and did not last, and was only done fully when considering promotion.

So, as you observe, now we have some new technique that managers are sure will increase performance by 50+%, if only people would use it. They can't just raise their expectations of performance by 50%, because they can't measure performance to within 50%! So, they measure the thing they can: token consumption.

That would be sensible if you were a car maker and not in the business of delivering posts. You’re tanking your core business, just to do what the developer of the tool should be doing (ensuring the usefulness and reliability of the tool).

I’m all for a trial run, but it needs to be done like any research experiment. With a goal and measurements along the way. Not by going blind and hurting your workers/customers.

How will companies like Uber continue to fund the “research” when the budget ends up burning 3x faster than predicted without being able to observe measurable gains?