-O3 also makes build times longer (sometimes significantly), and occasionally the resulting program is actually slightly slower than -O2.

IME -O3 should only be used if you have benchmarks that show -O3 actually produces a speedup for your specific codebase.