I actually read a really interesting article from a relatively small blog explaining that they're receiving a massive amount of scraper traffic from residential proxies.
replyThe article was called "The one we're commenting on"
> The LWN content-management system contains over 750,000 items (articles, comments, security alerts, etc) dating back to the adoption of the "new" site code in 2002. We still have, in our archives, everything we did in the over four years we operated prior to the change as well. In addition, the mailing-list archives contain many hundreds of thousands of emails.
replyDoes that sound like your typical self-hosted blog?
> Does that sound like your typical self-hosted blog?
replyWhen compared to
> They're most useful for getting information from the cloud hosted sites that hoarde most of humanity's output today like Youtube and Reddit.
yes, absolutely.
The Bright Data mentioned in the article, as well as other similarly malicious but even harder to identify parties, most certainly do fire up residential proxies, no matter what the site, no matter how useless or duplicate the data which they're trying to get. Not at first - they start by trying to get your content from cheap data center connections - but as soon as some kind of bot-mitigation appears, they move to residential proxies to try and evade that, with a first tier coming from "global south" residential proxies, and then scaling up to (presumably more expensive / less widely available) proxies from the USA (I've seen some from Europe, but very few in relative terms). Each tier also appears to have the option to run JavaScript.
reply