Intel published some guidance on doing precise benchmarks. Basically run your code inside the kernel, turn off interrupts, use the cpuid instruction to prevent out-of-order execution, use rdtscp instruction instead of rdtsc, etc.

The way you work around this (apart from doing what you can to make the system as predictable as possible) is to accept that the data is noisy, and then working around it by doing multiple trials and following up with a statistical analysis on the results.

You can use the desired confidence to inform the warm-up, number of trials, benchmark duration, and so on.

If you end up with a multimodal distribution it can be worth tracking percentiles.

Used to do this sort of thing for computations that needed to run in the 10 microsecond range (HFT stuff), circa 2008. Had very predictable results because:

a) language was not garbage collected (C++)

b) we avoided heap lock contentions in critical paths by pre-allocating object pools at startup

c) I/O operations were offloaded to separate threads, connected by mutex locked linked lists

d) processing thread was bound to its own CPU core

That's about as deterministic as we could get.