I think all of these mitigations are unfortunate. They hurt one of the things that makes the web cool: it's a stable, stateless, idempotent way to access data.

This makes it a prime target for aggressive scraping by LLM companies, but it also makes it accessible and fast, and a prime target for benign use (like archive.org or "read later" services).

For my own sites, I'll eat the cost of the crawlers (mitigated by making the sites as efficient as possible) and keep them available to everyone.