robots.txt seems like it should be a legally-binding terms of service which would make them outright copyright infringing.
replySue for $180,000 per infringement which should be calculated for each illegal API call.
Was your robots txt written by a lawyer? Does it hold up in the court?
replyIt doesn't matter. Robots.txt is not a license, it's a set of computer parsable directives of how programs should access your site. The actual license doesn't have to be written for computers to parse to be legally binding.
replyA person should be able to write in a terms of use or license page on their website that says "do not include any content from this website in your AI training data. if you do you will be billed $100 billion dollars." And it should be enforceable. It just turns out that nerds like to say "oh that would be too hard or too expensive, so we're going to ignore it."