upvote
ADAM is related if your second derivative matrix happens to be diagonal.

Of course, it takes about 5 minutes to show that any DNN is going to have very very high magnitude off-diagonal terms by the way it's constructed, so pretending that a diagonal approximation is close enough is crazy.

reply