The second moments of the gradient and the Hessian are absolutely related!

See the Fisher Information, and the Cramer-Rao Lower Bound (an inequality on how much the inverse covariance matrix and the Hessian can differ).

https://en.wikipedia.org/wiki/Fisher_information

I didn't find a direct proof earlier, just assertions - I've only seen it proved in textbooks that aren't linkable.

Theorem 1, section 1.3, page 2 shows that the expected variance of the gradient of the loss function and the expected second derivative of the loss function are equal at the minimum. I hate that the ADAM paper did not talk about this, this is something that is hammered into anyone who has taken a mathematical statistics course. This has been an established fact in statistics for well over 100 years.

https://courses.grainger.illinois.edu/ece563/fa2025/Notes11-...

Away from the minimum they can diverge, but there is a close enough connection to make it an extremely useful approximation.