One of the hardest parts of optimizations is not being penny wise and pound foolish. This sort of thing is a great example of that.

Additionally, a hard part is that all of this can change over time with new hardware! Some patterns that were crucial before everything gained branch predictors are irrelevant now, etc.