While dedicated scrapers/attackers could work around it, and they could do so much more efficiently than the client-side js, almost none of them do. Unless you like paying additional hosting resource fees to serve bots, it's a worthwhile option, and is less annoying to typical human visitors than cloudflare's interactive captcha/challenge which is what most people use.
The main author is aware that the algorithm is far from ideal for this purpose. See https://news.ycombinator.com/item?id=48869064 . If more bots start to answer the primitive challenge anubis uses now, that'll hasten implementation of a different algorithm.
Don't let the perfect be the enemy of the good enough. For now, the algorithm or challenge scheme almost doesn't matter. Since it's much smaller-scale than cloudflare's challenges, that's probably why very few scrapers and botnets bother to solve anubis's trivial sha2 pow.
Targeted attacks may not be repelled at all. That's not the point.
Checks for existence of a specific static cookie, if it does not exist, output a small page that sets the cookie via JS and reloads. Sadly this kills Noscript, but it would be possible to add a form in <noscript> that when submitted sets the cookie serverside.
Is this trivial to bypass? Yes. It still keeps out 95% of unwanted bots. Reality is most do not target you specifically they just want to mass-scrape with low effort. Running headless browsers is way more expensive for their op
I've extended this with a FCRDNS checked exclusion for Googlebot.
Another quite effective measure I figured out was checking the existence of Sec-Fetch-Dest header if the User-Agent claims to be a modern browser. If you don't want to close down too much.
Also, I only apply these rules to routes that are not cheap and cached.
For a webapp that sets a long-lived cookie, that cookie could be used to bypass anubis completely, or lower the weight in anubis so that it doesn't send its pow challenge unless there are major red flags. If bots start to abuse that exception, it can be removed.
They also don't load CSS but for some reason the security theater PoW won the mindshare.
Define "scale". For any reasonable wait that you're willing to impose on your users, any PoW scheme heavily favors attackers. They have unlimited time and can be scraping even while they're asleep. Your visitors on the other hand don't have that luxury. You might argue that's not the point and it's only to stop dumb scrapers that are effectively ddosing your site, but if it's just dumb scrapers, you could've stopped them less onerous measures like tls or javascript fingerprinting.
sha256(challenge + ascii(nonce)) means that the first compression round of the function can be cached and the second compression round is just the nonce plus the cache. This is the same trick used in Bitcoin mining and would have been avoidable by putting the nonce first, so immediately any non-naive code has to do half the proof of work as the vanilla solver.
write a implementation of the anubis proof of work in native c code, optimized for speed above all else. use every trick available to make the proof of work as efficient and fast as possible, including modern processor tricks on the x86 platform. your code should avoid using external libraries where possible, include tests, and be readable and concise. a reference for what needs to be met is in this repository. https://github.com/TecharoHQ/anubis
Then let’s develop this more. turn this solver into a local HTTP server that can be given work in the request, and it returns solved work. make an end to end tester that sends test work to the solver and waits for a valid response. add support for solving with a GPU using cuda.
Then it was done more or less, it happily made a local server that supports solving the challenges given to it in bulk with priority based queue and can tolerate potentially tens of thousands of requests a second with no issue. The CPU time spent solving the challenges is less than the SSL setup for the connections. The GPU version does in excess of 20GH/s (but with high latency) though I didn't really test it, I'm not using this for anything but proving a point that the LLM itself can write the bypass tools and run them happily.In addition, nothing prevents anubis from sending a wasm solver instead of js, reducing the gap between a custom native solver and a js solver.
The big blocker is that testing against dozens of googles chrome to ensure functionality on abandoned smart TV oses takes a nontrivial amount of time. As an example of the level of debugging and the like required: https://github.com/TecharoHQ/anubis/pull/1684/changes/67621f...
My office gets very warm when chromesweep runs.
This is something that is complicated enough that even though LLM tools can help, it's not a magic bullet. It's just complicated in general.
The more bots try to access the site, the more profitable it will be!