Can you do this for the general case? No, not even for extremely popular books. People might quote Harry Potter a lot, but they don’t quote the entire thing over and over, chapter and verse, on hundreds of thousands of different websites. The number of times Bible verses appear in the training data is going to absolutely dwarf the number of times Harry Potter quotes appear, and people aren’t quoting all parts of Harry Potter, just the interesting parts.
> When i ask chatgpt for a specific page or so from HP I get the impression that the model would be perfectly capable of doing so but is hindred by extra work openAI put in to prevent the answer specifically because of copyright.
They do put extra work in to filter this stuff out, but even if they didn’t the model wouldn’t be able to reproduce entire chapters, let alone entire books.
You can test this for yourself. Remember, this lawsuit isn’t against OpenAI, it’s against Meta. Download Llama and try to get it to reproduce Harry Potter. There won’t be any guardrails imposed on top of the model if you run it locally.
I'm fairly certain I could find the entire thing in plain text in multiple places online. A quick google gives the philosophers stone as the second result in pdf format on the internet archive but i'm sure with a bit of looking i'd bump into a lot of plaintext copies.
They might have taken measures to prevent this from being anywhere their training data (i think it would be fairly easy and something they'd likely do) but if they at any point failed for a book or so that they didn't consider wouldn't my original question stand?
The Bible isn’t just a book, it’s been a massive part of human culture for millennia, to the point of it shaping language itself. LLMs might be able to memorise the Bible, but it’s not because they can memorise books, it’s because the Bible is far more than just a book.
I doubt every part of those books get quoted everywhere on a numbered basis like the bible might be. For only recently public domain books it seems to be overly cautious trough the retroactively applied filtering where it refuses if it suspects there might be a single country where copyright still applies.
I decided to ask it: Can you give me the first 4 paragraphs of chapter 3 of the book The picture of Dorian Grey?
And it gave me something and it looked alright to me. It read right and i went to gutenberg and glanced over it and the first lines of each paragraph seemed correct but only the short ones were. The first paragraph which was longer after the opening lines suddenly had an entire section randomly replaced with hallucination.
A followup asking it to not hallucinate had it search the web to fetch the correct thing which isn't valid in this context.
I suspect it starts hallucinating once the bit of text gets long so i asked for specific sentences of chapters (and to do so without web search). the 1st, 2nd, 3rd and such.
It managed to not outright hallucinate lines then but did get the chapter i asked for wrong sometimes. I presume that with sufficiently careful prompting one can get the book out properly in sequential order with a lot of prompts but it takes quite some effort to get there. But that's where my curiosity ends for the night. My bed calls.
You failed to get it to reproduce one paragraph. Why on earth would you presume you can do it for the entire book‽