As someone who operates a large non-profit public data driven website, I have some VERY strong feelings about scrapers. We looked into various commercial solutions (Datadome, HUMAN) and based on our traffic estimates from logs we'd be looking at at least 250k/yr for bot mitigation. Anubis is offering a temporary reprieve, but after reading the recent kernel.org article [0] it's increasingly clear that this is a temporary bandaid.
replyThe cheapest solution is to require a login and rate limit by API key. I also have strong feelings about the tragedy of the commons.
I say put your data up in torrents, host a few KB of plain HTML linking to them, and let decentralisation do the rest.
replyNo strong feelings here is what I meant. Certainly, that energy is best directed into aggressive countermeasures and defense in depth of public goods.
replyhttps://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...