Hence why my solution is, ironically, micropayments. That would be so cheap that any human could even access thousands of webpages per day for pennies (or however expensive sending a signal across wires is…it can’t be much), including shared dumps of scraped data from billions of pages.
And Anubis: it is uncomfortably close to erotica, in a way that makes me panic when it pops up while I'm at the office. I wouldn't care if it was a picture of a puppy or something, but not an anime catgirl.
Equating bot prevention measures with “please give us permission for 1,000s of companies to stalk you around the Internet, no? Here are some dark patterns to try make you accidentally agree” is rather unfair IMO. They are an inconvenience but for the protection of the sites rather than for the betterment-of-adtech-via-conning-you-the-viewer-with-malicious-compliance.
> Bots can use async or virtual threads and not even have to hold on to a stack while they are waiting for this bullshit
They could, but given these methods seem to work currently most (almost all?) are not. They will start doing so if the amount of resources protected in this way increases to the point where it is worth it, or course, that is why memory-hard processes are being worked on which will make that route less practical.
> cookie banners, and of the that crap should be an automatic F grade on WCAG and should make web sites immediately liable for ADA lawsuits
I can't disagree there, though that would be quite an inefficient process and requires a bunch of money up front from who-ever champions the legal challenge, so I doubt it would actually be at all practical.
Don't the bots still need to do the proof of work for Anubis? They can't "async" their way out of that work.
> Another web is possible but web populists don't really believe it.
How? Sharing knowledge with the world used to be an obviously good thing. Now it's a way to help tech companies replace people with machines. What alternative web could change this state of things?
The implication is that while they are waiting for the async loop to complete for one site, the bot can be scraping others that have already had the challenge completed (usually the one challenge stays in force for some time afterwards) or that have no such challenge.
Memory hard challenges will reduce the effectiveness of this, but not remove it completely.
Though that these methods are working at all implies that most (almost all?) bots are not bothering to use such async methods as there are sufficient resources not protected by them that this complication is not yet worth implementing.
Sharing cookies along a strata of inter-team bots is one trick.
_
Web scraping is (among the other pieces) also for the more technically inclined among the "little people" to have computers do things that people they don't have the money to hire might do. Making web scraping more difficult, which won't stop the big players who will work around the hurdles one way or another, is "pulling up the ladder".
_
> What alternative web could change this state of things?
A different architecture for trafficking web contents, one that doesn't need the "shield" of something like Cloudflare or "tar pit" of something like Anubis in the first place.
IPFS is or is something like one piece of that picture but there are other essential pieces to complete that picture.
I know I'm incensed at all of this because I feel like I'm the one that endlessly waits in front of the Anubis challenge for 5-10 seconds, or keeps getting extremely difficult or brain-draining hCaptchas thrown at me with no real solutions.
And I'm stuck with low-end compute for probably the rest of time at this point - not a single machine in the house has more than 8GB of RAM. So I'm just doomed to wait and burn my time and my sanity some more.