They are poorly implemented by the "fuck you I got mine" crowd. They will get stuck doing things like trying to run through a calendar that could theoretically go back to the beginning of time and all the way to the end of time. And because that calendar might change, it gets scraped for every inquiry made to the poorly implemented AI system.

Similarly, I've never understood the economics behind the constant rescraping that is flooding the internet or really what's triggering it.

It can't all be agents reacting to user queries. It's confounding how much CPU and bandwidth is getting flushed down the drain.

My laymans understanding is that poorly written scripts get executed and then owner comes back occasionally to check that there is 'content' in their database. They're not sitting there troubleshooting their thing beyond 'it's getting data' because, for similar reasons to their scripts being poorly written, their strategy is 'get data'.

Nuance, complexity, and an awareness of 'other people' do not exist in their worlds.

The most recent data on the internet for advertising, intelligence, etc.

And a lot of bad scrapers.