upvote
I still think it is just a matter of time until scrapers catch up. There are more and more scrapers that spin up an full blown chromium.
reply
It seems inevitable, but in the mean time, that's vastly more expensive than running curl in a loop. In fact, it may be expensive enough that it cuts bot traffic down to a level I no longer care about defending against. Like GoogleBot had been crawling my stuff for years without breaking the site. If every bot were like that, I wouldn't care.
reply
Cool, if they're running full blown chromium maybe the next step can be mining bitcoin on any pages served to bots.
reply
Even that functions as a sort of proof of work, requiring a commitment of compute resources that is table stakes for individual users but multiplies the cost of making millions of requests.
reply
AFAIK you can bypass it with curl because there's an explicit whitelist for it, no need for a headful browser.
reply
Well it's a race, just like security. And as long as anubis is in the front, all looks bright
reply
> Turns out careless, unethical vibecoders aren't very competent.

Well they are scraping web pages from a git forge, where they could just, you know, clone the repo(s) instead.

reply
"Turns out careless, unethical vibecoders aren't very competent." well, they rely on AI, don't they? and AI is trained with already existing bad code, so why should the outcome be different?
reply