For the curious, eigenvalues only exist for square matrices. Singular values are like generalized eigenvalues.
replySingular values are like the fundamental frequencies of your matrix. You know how you can define any color with RGB? In a (pretty handwavy) way, singular values are like RGB color codes for us math guys.
Optimizers like Muon and Adam play around with weights' first, or second order singular values to train models.
Last statement is a bit sus... Muon computes matrix sign function which can be defined as setting singular values to 1, though you can also define it without SVD. Muon itself doesn't use SVD. Adam doesn't do anything related to SVD or singular values. Also not sure what you meant by "second order singular values"
replyJust going to sound really pedantic here, but RGB does not capture the entire colour space. In fact, it only captures about 35% of the colours the human eye can perceive.
replyhttps://www.oceanopticsbook.info/view/photometry-and-visibil...
You seem to be conflating "RGB" with one particular RGB color space: sRGB. That's a common enough conflation to make, but not appropriate when you're trying to be pedantic.
replyOkay, but that was a really useful metaphor if incomplete in a lot of ways. It made me say “oh”.
replyThe SVD seems to come up everywhere in my work in computer vision. I find myself continuously using the various C++/Eigen SVD implementations. Actually I should speak in the past tense. Claude and Codex are now generating all my code for me now, and I see them spitting out SVD code frequently -- often for very special cases. SVD truly is an amazing tool.
reply