The thing is, I explicitly wanted to not focus on current legality, because I believe this is leading us astray. This is about how we want to shape our society, not about how their actions fit into a legal system that wasn't designed to handle AI training. If, by chance, all of this was legal, the same arguments would still apply. For example, see how Adobe is training their image generating AI on licensed work. The result -- privatized intelligence -- is the same as when illegitimate training data would have been used. Or, say, an AI company makes a deal with all the publishers to license their books for training. Or a court decides that training AI with MIT licensed code is totally fine. Or an AI company pays legions of software developers to train their AI. All of this still ends in a power grab. All of this still privatizes knowledge and skills accumulated over centuries, if not millennia.

>Fair use law isn't supposed to apply to commercial activity.

No, otherwise there would be a straightforward "non-commercial" clause. Instead there's a 4 part test, which takes usage (commercial or not) into account, but doesn't hinge solely on it.

https://en.wikipedia.org/wiki/Fair_use