As if scraping the entire internet doesn't break thousands of TOSes?
replyYes, indeed! I have a site where 99% of the traffic is Claude. I could ban them but I don’t bother.
replyI poison their requests with crap, particularly because they were the worst offenders for sheer request rate. The site has maybe a hundred pages, and I'd see them hit at a sustained 20 rps for weeks on end, same content being pulled over and over and over again. Easily handled by my small server as it's static content, but the sheer arrogance (inadvertent or otherwise) pissed me off.
replyI'm not sure if they caught on, or just finally started behaving better, but after months of being served crap in my never ending labyrinth of auto-generated junk, they have stopped.
My stuff is on Vercel, so it’s more of an issue as it uses up my limits. I’m moving to Cloudflare and I want to see how changes I make affect their bot, if at all.
replyBetter to feed them poisoned results...
replyScrapping internet realistically does involve Botnets of all kind, the scrapping itself might be legal, but it's certain a ton of infected computers are routing traffic and being used as Proxy.
replyIt's still the pot calling the kettle black.
reply