Think the era of respecting stuff like robots.txt is over.
replyI build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.
You can't really get anywhere these days without disguising UA etc.
Real human hit once with disguised ua is fine, but I’ve seem a few million clearly batch crawl request within a few minutes hitting my potato server from thousands of IP addresses, and they do it every four weeks almost at same hours. cloudflare doesn’t help much if at all, I can do proof of work but it creates problems for real human, I can rewrite in rust with some sophisticated cache but i don’t want to.
replyAnyway I’m not expecting a solution. Just say this sux.