It's telling LLMs how to download all their files in a way that has the least impact on their infrastructure, while telling it that any other way will be met with CAPTCHAs. In the short-term, that seems beneficial. LLMs can be quite persistent in their bad crawling attempts

What the role of Anna's archive plays in the future is an interesting question. But I'm optimistic about it. And if Anna's archive fails, but lots of OpenClaw instances are hosting the torrents or at least have a local copy of parts of the library that's still a decent outcome

They are trying to distribute information, not get traffic.

The hope is probably that the LLM's will download properly rather than DDOSing them.

Honestly I think they are being a bit naive and assume that the scrapers gives a shit.

A few of the large AI companies might care enough to set up a custom solution for you, assuming that your dataset is sufficiently large. Most doesn't. HTTP is the common protocol and HTML the standard format, a torrent is just needless hassle.

The problem Anna's Archive also have is that the legality is questionable and having an official collaboration with them might be problematic. Better to just crawl the site and claim that you crawl the entire web so you accidentally crawled Anna's Archive.

> Why would they tell the LLM exactly how to download all their files in bulk for free? Isn't that the opposite of the self-preservation they're trying to do?

The goal of AA is to spread the data for free, not to gatekeep it. Donations are optional.