It's fascinating how intuitions differ. To me, it doesn't feel like theft at all. For one thing, theft is depriving another of something, and has therefore never been a good metaphor for infringement; hackers used to be the most insistent about this principle, and it's weird to see a doctrine which was cooked up in a literal AI lab get thrown out the window for literal AI.

But pretending you said "infringement", for me it comes all the way back to the Constitution: "To promote the Progress of Science and useful Arts". I cannot possibly twist the development of large language models into something which violates the spirit of that purpose. I don't see how anyone can.

Your point about the scale is valid, and the alienness of it, sure. But you haven't made the case that the vastness of the scale should affect the conclusion.

Something I left out in the first post is that copyright is meant to protect expression, and not ideas: this is the deciding factor in the 'nature of the copyrighted work' test for fair use. More expression, more protection: more ideas, less.

I think the visual arts have a strong case that image generators directly infringe expression: I'm not convinced that authors do, and I think software should never have been protected under copyright because the ideas-to-expression ratio is all wrong for the legal structure. There's clearly no scale case to be made for ideas: "but what if it's _all_ the ideas" fails, because the ideas are not protected at all. Nor should they be, that's what patents are for, and why patents are very different from copyright.

LLMs are remarkably good at 'the facts of the matter', hallucination not withstanding. They're very poor at authorial 'voice transfer', something image generators are far too good at. It's when I start asking myself "well what even _is_ this 'expression' thing anyway?" that I conclude that we're out over our skis on the LLMs-and-IP question: precedent can't tell us enough, and that leaves legislation.