I had meta's crawler hitting the pi searcher at something like 5qps for days on end, just ... querying for substrings of pi, ignoring robots.txt, etc. it wasn't enough to break anything but it triggered a lot of alerts.

I can imagine that sites with dynamic content and potentially unbounded query types or pathnames are in danger from particularly stupid crawlers.

I always thought it's the web search tool.

Grok actually shows a number of sources used for an answer. Once I asked it something simple and it apparently scanned 200 different websites. And it was just a short prompt. Now imagine millions of users asking for something multiple times a day.

I kind of wish the recent Google monopoly court ruling had forced Google to open up their index to anyone, not just Perplexity/other big players.

> a badly configured scraping setup?

Cynical-me assumes every single AI company is vibe-coding everything, and _all_ their scrapers are as badly written as the typical publicly available scraper code and tutorial - mostly written by self promoting spammers and SEO "experts" in the late 2010s.

Any they all DGAF about wasting website owners server/network resources, of the CPU and network resources of the "dumb schmucks" who have a free vpn installed or a factory-hacked cheapo media box or mobile game the developer has surreptitiously monetised with a residential proxy sdk.

It also wouldn't surprise me at all to find there are dozens of competing training data acquisition teams at every frontier and wannabe frontier AI company - scraping the entire web in parallel to meet internal KPIs. Half of which have lost entire datasets due to vibe coded storage and archive setups.

Maybe they have just too much money at hand, would not surprise me, people are still investing into gen AI like there is no tomorrow. Also, for the completely criminal operations, you only have to find a way to infect and distribute your bot to, e.g. some common internet of shit device. Scaling is basically free afterwards as you don't need to ask anyone. The article also hints that those are actually the biggest problem.

Then there is probably also a lot of time pressure on the people implementing and operating those scrapers so they have even less incentive to optimize their code.

It should really just be called DDoS, at a certain level of incompetence intent doesn’t matter. You’re right that there’s zero (information gathering) benefit over reasonable scraping which wouldn’t cripple the site.

Who’s doing it, are they even using the data?

750k items in their content management sysem. N independent labs crawling wanting to check that every day could easily give bursts of millions per hour

Millions per hour is tens per second though; perhaps the fix is performance improvements

Maybe someone is paid per scrape, without reduction in payment for duplicates.

Maybe every web query for Linux commands in $LARGE_COUNTRY checks all the Linux websites again.

Hah. I have a homelab with a couple of sites, including a personal Forgejo installation.

Last night my server turned off because it went into thermal protection shutdown. Turns out, my all-in-one cooler has inoperative fans, which I normally never really notice. The passive heat dissipation from the water cooler is more than enough.

However, this time they hammered my computer for 12 hours with about 200 requests per _second_ to my Forgejo.

Maybe they are aggressively scanning for updates on the page

It's not AI companies scraping these websites, it's AI companies creating a massively profitable need for data, and every random Joe with vibe coded scrapers tries to make a buck out of it.

It isn't. AI scraping has nothing to do with it, for the reasons you said. Someone wants the web to go offline, they are DDoSing the entire web, and it's working. For some reason we are tackling the symptom instead of finding out who that person is. Come on, it can't be that hard to subpoena Bright Data. The law enforcement system knows how to track down someone who's trying to be anonymous on the internet.