I find 98% of the time that users are clamoring to get something implemented or fixed which isnt speed related so I work on that instead.
When I do drill down what I tend to find in the flame graphs is that your scope for making performance improvements a user will actually notice is bottlenecked primarily by I/O not by code efficiency.
Meanwhile my less experienced coworkers will spot a nested loop that will never take more than a couple of milliseconds and demand it be "optimised".
Also the rule (quote?) says "speed hack", I don't think he is saying ignore runtime complexity totally, just don't go crazy with really complex stuff until you are sure you need it.
People don't ask for software to be fast and usable because it obviously should be. Why would they ask? They might complain when it's unusably slow. But that doesn't mean they don't want it to be fast.