However, the refrain from naysayers isn't that there's too much enthusiasm. It's that anubis doesn't make sense as a theoretical barrier against bots. Who cares as long as it works empirically, even briefly? Are you suggesting people let their sites continue to be degraded or unusable until they find a better solution, even though PoW challenges can improve the situation today?
Not everyone has the luxury of writing their own web app designed to handle massive bot traffic at minimal cost, gutting functionality to meet that requirement if necessary.
I noticed a couple of weeks ago that 1-2 orders of magnitude more bots were getting past anubis challenges. However, it would be incorrect to conclude that it's doing nothing and now worthless aside from heating up visitors' cpus. If I turn it off, I see >2x bots.
The next salvo in the war of attrition will probably be when anubis switches to wasm hashx, which the author suggested, in previous HN threads, is on the roadmap.
If bots quickly adapt (Claude vibe-coded JIT wasm-to-native-transpiler for hashx challenge algorithms?), that may be the end of anubis and efforts like it. If they don't, I expect more HN threads with people complaining that such compiler efforts are possible and therefore PoW-gated sites are doing it wrong... even if those sites stay usable.
I'm just saying, it's bad computer science.
A better, perhaps the best, argument is that a worse bot mix might appear tomorrow, so anubis should be used as a runway for a better solution, and not as a solution in itself. If only everyone with a website had the desire (or AI tokens) to implement their own better solution. In some cases even desire or a lot of tokens isn't enough. I mean, you think kernel.org folks haven't put some serious thought into it? And yet, as HN argues about it, there's kernel.org running anubis at difficulty 5.
It shows they really don't care and they are just rage-baiting readers in their weekly grumpiness without any suggestions.
Unless they come up with said "solution", they can completely ignored in their complaints.
So could you give a short elaboration on what was this "better computer science" solution that you came up with in your discussion?
- It's the SOTA companies doing it?
- Scrapers are doing it for training data?
Those are two assumptions I see in posts and threads around Anubis, that are taken at faith, and never once substantiated.
[0] https://www.ft.com/content/07611b74-3d69-4579-9089-f2fc2af61...
There are more than just the American top dogs (OAI, Anthropic, SpaceX, Facebook)... especially the Chinese government with all its infinite cash resources and next to zero ethical constraints.
I don't trust the US top dogs at all, but I think the fear of discovery alone would lead them to not use "residential proxy" services. Non-US/EU entities however... who cares?
The scraper problem now is fleets of residential proxy devices - often things like smart TVs, phones, and browsers with some “proxy SDK” installed as part of an app’s monetization scheme. They make a couple of requests to a site - just enough to fly under the radar - and move on to a different site.
If each new site they hit forces them to solve a proof-of-work, that’s a meaningful dent in their scraping performance. Many of these boxes may not even have the spare CPU power to efficiently solve so many proofs of work - and anything that makes an owner notice their device is running slow is something that could meaningfully impede adoption of these SDKs, or force the operators to choose between minimizing performance impact or scraping more sites.
I just checked, and these many months later after installing Anubis, I'm getting about 14K requests per day. It's unreasonably effective at cutting crawler abuse. It should not be able to reduce my traffic by 98%, and yet in practice that's exactly what it's doing.
I think that the most abusive scrapers are operated by people who absolutely suck at programming. If they didn't, they wouldn't be crawling every permutation of "show me the git blame for every file in this repo for every commit in its history". Were they actually competent, they'd be investing in smarts such that they weren't crawling a web frontend to get that information.
See also how captchas continued being effective for years despite services like anti-captcha offering to solve them for you for a fifth of a cent each by farming the work out to India. It took advances in AI that made it viable to reliably solve them on-device to bring the end of the captcha
I don't think that's generally how these networks use residential exit proxies. There are at least a dozen well-developed frameworks out there for decoupling the crawler from the network exit point.
Most res proxy exits are just slinging bytes for clients using SOCKS, or another tunneling protocol.
If nothing else, a modern scraper will want better control over their TLS fingerprints, and you can't get that if you're depending on the on-device TLS libraries alone.
All they need to do is forward traffic and launder its IP origin. They don't even need to be able to (and would actually be well advised not to) decrypt TLS streams.
Until they discover this neat trick [1] and solve challenges orders of magnitudes more efficiently than legitimate users.
The game theory of Anubis is not sound. It makes fundamentally less sense than Captchas, and even those have been on the way out for a while.
Evidently, it makes _enough_ of a difference for the people running it. Criticising it is just letting the perfect be the enemy of the good.
The alternative to Anubis isn’t “something better that isn’t being picked”, it’s “completely unprotected”, and the people criticising it for being imperfect could helpfully move the conversation forward by actively suggesting a concrete alternative.
> meaningful dent