On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
Project is here on Github: https://github.com/UninvitedActivity/UninvitedActivity
Grub was, in a very real way, a botnet. And, we harmed site owners when we were operating at full capacity. There were a few bugs in the early days where we would reschedule a site because the ingestion in the server broke, which then caused the page to be rescheduled. Stupid error, and we fixed it, but it's illustrative of the fact even good intentions isn't enough here.
What I've come up with over the years is similar to the idea Cloudflare is implementing with payments to site owners by charging the crawlers. My objection to Cloudflare's implementation is based on a personal opinion about Cloudflare being a single point of failure and also a decrypted choke point. Their ideas about how to handle crawlers, and pay for the load on the sites is solid. It presumes to use the 402 response to demand payment. I'm clearly biased about Cloudflare, but that's my prerogative here.
It may be possible to solve this with cryptocurrency, in a distributed way, and I've prototyped a system that uses the Lightning Network to handle the payments from a 402 response. Lightning Labs also worked on a project called Apeture for a time that did something similar.
HN's site knows every item ID, and it knows fresh IDs get read in a predictable distribution while old ones mostly sleep. Sustained access outside that is itself the scraper signal. No IP reputation needed, which matters now that residential proxies burn an address after a handful of requests.
Karma gives you a clean way to let humans through. Issue logged-in accounts with decent karma a token whose cold-content budget scales with it (the karma), so an account with history scrolling back through a 2014 thread just reads it. Karma should gate the tier, not be spent as currency, or upvote rings become a crawling business.
Anonymous readers who deep link into one old thread from a search engine get the first fetch or two free (and you watch the article IDs, not the IPs). What remains after those carve-outs is bulk traversal of cold IDs with no identity attached, and that traffic gets rate limited and answered with a 402: pay per page over Lightning, priced at a healthy multiple of what residential proxy bandwidth already costs, or come back slowly for free.
There are probably holes in these thoughts. It's one of the harder problems to solve, for sure.
Is that a side effect of whatever you are doing?
There are other reasons why we might not respond, but overload is the main one.
Edit: I only see one email in the archive related to your account. It was from Sept 2024 and we responded to it. Are you talking about a different account?
It's indeed from two different email addresses from the one on this account. They're not that important so never mind, thanks for checking and for the reply!