The only reason at this point that there is no way to prove the claim is true is that we cannot inspect the weights, nor query for copyright violations.
I imagine that someone will eventually figure out how to query black-box LLMs for copyright violations without triggering their guardrails. This is an existential threat for anyone running a black-box LLM with stolen data.
We already know how to and have proved it, before the guardrails got "better" (mostly at hiding it). Early versions of Claude and ChatGPT were fed with a paragraph or two of popular fiction and spat out a half dozen paragraphs more, sometimes verbatim. They're significantly lossy compression algorithms. Not judging just based on that, but it's clear that their creators engaged in far more than the already outrageous amounts of copyright infringement that they've admitted and been so gently slapped on the wrist for.
This is trivially true. Why can they work with a tool or API I’ve written or something internal to a company? They definitely have not trained on my local machine from 2020.