undefined

points

[-]

I would probably dig into the reasons for the differences in the benefit on the test machine and in prod

I had an issue like this for optimizing pgrust. I had an optimization that showed no impact on my test machine (c8g.4xl) and showed a 20% improvement when ran on my mac. It turns out the issue was the instruction cache on the c8g.4xl was being saturated on the test machine but not on my laptop, moving the bottleneck to a different place

If you can consistently reproduce the performance difference, you're already half way there

by marginalia_nu11 hours ago|

parent|

[-]

I'm pretty sure the reason for the difference is that production machine exists in a state of mixed memory residency and low grade resource contention that is incredibly hard to replicate in a test scenario (as the moment you start making queries the pages warm up, and the test becomes unreliable).

The hard part about optimizing this type of code, IMO, is that there are so many cache layers, both in the CPU and the OS and sometimes in the storage medium. You can warm all of those caches up, but then you're testing a nonsense scenario that will basically never happen in a realistic scenario, where wall clock time is what matters, and not much the CPU is working or how many IOPS you're pushing.

by postgresperf4 hours ago|

prev|

[-]

Optimizing a multi-layer cache system is a Complex System task with no subset you can reduce to a simpler version of the problem. PG has a 3 layer cache--database, OS, storage--and when you combine that with concurrency and locality, you have a giant 5 dimensional space to map out. Building the right set of benchmarks for PG with guidance for the most important parts to navigate fro that space has kept me busy for 20 years now.