Very confused by this comment. The older (poorer) parts of the ML literature focus on models with convex and (gradient-)Lipschitz objectives, but that's not representative of reality, not even close. Modern objectives for AI models are famously nonconvex (catastrophically, from the point of view of classical optimisation theory), and that's where the interesting research is.
replyI'd push back on this. Most of the core optimization techniques (eg, ADAM, stochastic gradient descent) are straight out of the convex optimization literature. Generally you need to use optimizers that work well on convex objectives because near minimizers, functions tend to be convex. (Proof by contradiction: a non-convex point has a strict descent direction.)
replyThe fact that neural networks are highly nonconvex has encouraged a lot of research, but it's more of the kind aimed at resolving tension: these methods are probably good for convex functions, why do they continue to work for nonconvex problems, and are there tweaks we can make to improve them in that setting? It's not a lot of de novo theory; more standing on the shoulders of giants, etc etc.
The optimizers are lifted from convex optimization, but the point above was that they are applied to highly non-convex problems. They work for finding local minima, but a lot of the deeper literature does not translate (e.g. the conjecture being discussed in this post).
replyADAM does not work on simple convex problems [1].
reply [1] https://parameterfree.com/2020/12/06/neural-network-maybe-evolved-to-make-adam-the-best-optimizer/
[2] https://arxiv.org/pdf/1905.09997
I'll point out that "does not work" is not the same as "not as efficient" :) But it does seem the Adam paper had an error.
replyI think that Nesterov's first order method is the most efficient general first order algorithm on convex problems, so anything else is in some sense worse. (Edit: removed incorrect ADAM comment.)
Yours' "not as efficient" in [2] means that, sometimes, ADAM "does not work." Look at figure 2, ADAM literally does not work in the case of "true model."
replyYes, apologies, I didn't read the articles you linked before posting this. I did update the comment.
replyI don't think this changes the point, which is that most optimization methods used in AI owe a substantial intellectual debt to convex optimization theory.
Another intuition is that near a minimum you can Taylor expand the function and show that the higher order coefficients (past the square) are negligible.
reply[flagged]
replydeleted
replyWhat do you mean by this? A neural network hypothesis space is not typically strictly convex or a lipschitz function.
reply