I'll point out that "does not work" is not the same as "not as efficient" :) But it does seem the Adam paper had an error.
replyI think that Nesterov's first order method is the most efficient general first order algorithm on convex problems, so anything else is in some sense worse. (Edit: removed incorrect ADAM comment.)
Yours' "not as efficient" in [2] means that, sometimes, ADAM "does not work." Look at figure 2, ADAM literally does not work in the case of "true model."
replyYes, apologies, I didn't read the articles you linked before posting this. I did update the comment.
replyI don't think this changes the point, which is that most optimization methods used in AI owe a substantial intellectual debt to convex optimization theory.