There's a difference between someone running a scraping tool occasionally and bots constantly and rapidly re-scraping the same site over and over again
replyThe author's website is responsible for storing its own data. AI services currently treat the entire web as their storage and cache layer.
replyThis is an important context: the site is more likely to be targeted by scrapers because it is a curated collection of scraped information.
replyDo the bots care? Seemingly very little intelligence in many of them. Could be as simple as the site has more pages, so more traffic.
replyLoot first, ask questions later.
A curated collection of people who give away money. Was ever sweeter honey ever found in a pot?
replySimilar to the dose making the poison - the thing that jumped out at me in this blog was the ratio of scraping to visits. Unless OP is scraping thousands of times a day I don't really think they're in the same class as the bots they are blocking.
replyOP here: I'm not scraping thousands of times per day! Usually just a few times per year.
replyBut you could just be one of thousands of bots targeting the same sources you’re scraping.
replyWho scrapes the scrapemen?
replyLive by the scraper, die by the scraper
reply