> I’m skeptical that the problem they are trying to solve is truly unreasonable bandwidth demands.
replyNot necessarily bandwidth demands so much as processing demands. Scrapers have a tendency to hammer on parts of web sites that are computationally expensive to generate - e.g. search results, diffs and blame views in git forges, sorted/filtered/paginated lists, etc. Ordinary users may click a few of those links for things they want to see; scrapers will try to request all of them, even when 99% of them are redundant.
What’s more, they will scale up with increased resources on the site.
replyIf you redline at 20 searches a sec, and put in 4 more workers, suddenly you’re serving 100r/sec to the bots, paying 5x for it, and your users are still seeing shit qos. I've seen multiple cores of nginx saturated just dealing with one dos/crawl run on a somewhat high profile site.
I don't think people sit around going "Grrrr who can I ban next?". Instead this stuff gets noticed because you see the webserver at 99% CPU utilization for 2 days straight, check the logs, and see you are somehow getting crawled by half the IPs in New York City.
replyIf it weren't a real problem, these types of articles and services wouldn't exist.
replyPlenty of complaints exist about things that are not real problems.
replyWell that's not what's happening here, lol.
replyWhy the air quotes? Evading a ban and using (potentially ill gotten) residential ips to circumvent that refusal of service, is a bad actor.
replySurely that depends on the motivation for the ban.
replyNo. When someone says you’re not welcome, you’re not welcome. Regardless of the reason.
replyIf I get kicked out of Epstein island because I refuse to **** a child, that doesn't make me a bad actor.
replyAnd then going there 1 million times under fake identities? Yeah, I'm sure that's not a bad actor.
reply[dead]
reply