I think this is what will happen. That the public internet will become the place you go to seed the data you want to the scrapers and you will use a private internet for everything else. Private sites, private feeds, mesh networks, etc. We're basically going back in time similar to when AOL and friends had their own private networks for their members.
replyThe only answer is WebDRM.
replyIt's easy to pretend you're human, it's hard to pretend that you have a valid cryptographic signature for Google which attests that your hardware is Google-approved.
Crawling is the price we pay for the web's openness.
It's not hard to bypass attestation, it's actually very easy and done right now at scale, there's giant click farms with phones on racks.
replyThey don't modify any device and will pass whatever attestation you try to make.