Once I got very confused when exactly this kind of 10k x speed up I demonstrated for a problem introduced to me as a business bottleneck then got ignored (I wasn’t on this team, though, but it’s still hard to understand the mentality and incentives there).
My rule of thumb is that as long as it doesn’t really complicate or convolute the code, write with performance in mind.
My expertise is in C# and sometimes this is as simple as using a hash set or dictionary instead of a plain list.
Other times it’s having a decent understanding of how EF (an ORM) actually works (the static typing benefits of them in C# makes ORMs very valuable when used appropriately) and the cost of round trips to any network host (including a DB).