I'd be very interested in AI capabilities in detecting anomalous activity given that it should be very effective at pattern matching across large data sets. What would be required, however, is significant training with industry/application specific 'normal' browsing traffic patterns to determine acceptable tolerances, and then design escalating deterrence if and when tolerances are breached.
How is the scraper process stepping through the site? It must be programmatic, even if it's programmed to be random. What does that look like in traffic data in terms of timing and content requests?
What hazards can be put in place that a normal user would never go near, but a scraper is likely to trip over?
If scraping is being performed from multiple locations in parallel, what might that look like? ie. joining multiple sessions together that may show a more 'scraper' kind of behaviour.
Under IPv6 wouldn't detection start burning a lot of addresses? Yes, the pool is large, but that just means it's a scaling issue from IPv4 to IPv6, rather than a new problem requiring new thinking to solve? (assuming ways of detection were successful).
I'm being somewhat devil's advocate in my blocking rhetoric. The internet is in a weird confluence of wild-west and surveillance state. I like that there are ways and means of being anon-/pseudon-ymous; I like a bit of anarchaos, that's where the spirit of hacking lives. But I also don't want it sticking to my boots when I walk in the house. I'm also not fundamentally against scraping; there's definitely good scraping. There is an irony, though, in protecting scraped data or services that are built on scraped data (and I'm not accusing you, personally, just my general logical ideology of scraping). On residential proxies, I think they're 99% run by erstwhile criminals leveraging what were, until recently, referred to as botnets into a service they can launder their botnets through as 'I am legitimate business man'. I think most, basically all, residential proxies are run involuntarily or unwittingly and that blocking them would provide a greater society benefit than it would a drawback. Anything that promotes the sanitisation of end user devices can only be a good thing. The whole ecosystem is in a terrible state and regulation doesn't even seem to be on the distant horizon.
Maybe I'm hypocritical, or I'm trying to maintain mutually exclusive positions. This is where I'm at though.