On my external-most device I have a firewall that logs addresses that attempt to connect to ports behind which there are no services, and therefore there is no reason for the existence of that traffic (at least as far as I'm concerned), and therefore I treat it as malicious.
The address is recorded and goes into a database.
Periodically, the database is dumped to a file in a format that the firewall reads, and all the 'malicious' addresses detected above are added to a list so that those addresses are blocked from accessing the legitimate service ports. (analogy: if you throw an egg at my outdoor wall, I'm not going to let you into my house through the door because I don't want egg on my furniture).
I have a blocking period of about 3 months - because the things I run are important to exactly a single person. A blocking period much shorter would be recommended to prevent the gross-overblocking of legitimate users who may have un-lucked into being assigned a residential IP address that was previously used in a proxy-scan-scam.
Discard this if it's a stupid idea at-scale, but I quite the like the 'idea' of it, and I made it work, mainly for the technical challenge.
Project is here on Github: https://github.com/UninvitedActivity/UninvitedActivity
Grub was, in a very real way, a botnet. And, we harmed site owners when we were operating at full capacity. There were a few bugs in the early days where we would reschedule a site because the ingestion in the server broke, which then caused the page to be rescheduled. Stupid error, and we fixed it, but it's illustrative of the fact even good intentions isn't enough here.
What I've come up with over the years is similar to the idea Cloudflare is implementing with payments to site owners by charging the crawlers. My objection to Cloudflare's implementation is based on a personal opinion about Cloudflare being a single point of failure and also a decrypted choke point. Their ideas about how to handle crawlers, and pay for the load on the sites is solid. It presumes to use the 402 response to demand payment. I'm clearly biased about Cloudflare, but that's my prerogative here.
It may be possible to solve this with cryptocurrency, in a distributed way, and I've prototyped a system that uses the Lightning Network to handle the payments from a 402 response. Lightning Labs also worked on a project called Apeture for a time that did something similar.
HN's site knows every item ID, and it knows fresh IDs get read in a predictable distribution while old ones mostly sleep. Sustained access outside that is itself the scraper signal. No IP reputation needed, which matters now that residential proxies burn an address after a handful of requests.
Karma gives you a clean way to let humans through. Issue logged-in accounts with decent karma a token whose cold-content budget scales with it (the karma), so an account with history scrolling back through a 2014 thread just reads it. Karma should gate the tier, not be spent as currency, or upvote rings become a crawling business.
Anonymous readers who deep link into one old thread from a search engine get the first fetch or two free (and you watch the article IDs, not the IPs). What remains after those carve-outs is bulk traversal of cold IDs with no identity attached, and that traffic gets rate limited and answered with a 402: pay per page over Lightning, priced at a healthy multiple of what residential proxy bandwidth already costs, or come back slowly for free.
There are probably holes in these thoughts. It's one of the harder problems to solve, for sure.
Is that a side effect of whatever you are doing?
There are other reasons why we might not respond, but overload is the main one.
Edit: I only see one email in the archive related to your account. It was from Sept 2024 and we responded to it. Are you talking about a different account?
It's indeed from two different email addresses from the one on this account. They're not that important so never mind, thanks for checking and for the reply!
I did an experiment and linked from HN to my lame blog site and disabled all my anti-scraping measures. Even with all the bots I did not see that much traffic. I suspect some people are specifically being targeted by very poorly configured or very poorly written archiving scripts. Just one example thread discussing this with someone on HN [1]. Each case of being targeted will require looking at generalized characteristics but most are easy to stop in my opinion and experience.
<https://news.ycombinator.com/item?id=32048148>
<https://news.ycombinator.com/item?id=32031243>
(AFAIK that specific failure mode has in fact been addressed.)
I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).
Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.
Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.
I reported on findings at the time and several times since:
<https://news.ycombinator.com/item?id=36078578>
<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>
I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.
API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.
That said, a standard, say, JSON packaging of website contents available on request might be fun to have.
You're right to point out that if you're trying to get the contents of dead objects the API is of no use though.
On dead/flagged items, there's some value. Whilst the title/URL context aren't available, just knowing what fraction of submissions and comments are moderated is interesting data, and it is possible to construct patterns against specific accounts.
I'm frequently encountering what appear to be banned accounts. Being able to trace those through the API to see where and when they were banned, or now much moderated activity they're generating, can be useful. I'm relying heavily on the "/replies?id=<UID>&by=<moderator>" search endpoint (generally dang, tomhow, sctb, or pg as mod) currently to find out if there was a specific ban admonishment from a moderator. That's often but not always the case.
But it's not possible, say, to tell through the API what sites are banned. Looking at site history with "showdead" enabled can tell you that though, e.g.:
<https://news.ycombinator.com/from?site=synthetica.cloud>
(From the New queue, one of several "dead" submissions not flagged, suggesting a site ban.)
Hypothesizing an undocumented "site" API endpoint ... doesn't seem to check out:
https://hacker-news.firebaseio.com/v0/site/synthetica.cloud?print=pretty
not found
And there's the "day" endpoint doesn't seem to work either, though it conspicuously does not report "not found", e.g.:
https://hacker-news.firebaseio.com/v0/day/2025-07-10?print=pretty
... it's starting to make sense, but ...
... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:
<https://news.ycombinator.com/front?day=2008-11-05>
It's the collection of 30 items from that date I'm interested in. For my scraping, I don't actually need to further query the individual posts as I've got the elements I'm interested in (title, date, story position, URL, votes, comment count, submitter, site/domain) from the index page itself, parsed out of the HTML. The "Past" entries alone are a significant (though not huge) request load. To update the past three years would be about another 1,000 requests, which, if fulfilled and modestly rate-limited would hopefully not keel the servers over.
Once I've pulled in those "Past" pages, I could of course do further API queries, though at this point I don't see any specific need to do so.
I suppose that requesting the "past" links be included in the API set could be a request I might make of HN, or the ability to request, say, all submissions (or comments!) for a given date.
There are groups which have done HN analytics in the past using the API, for example Whaly.io:
"A Year on Hacker News" <https://news.ycombinator.com/item?id=31295219>
"Top Hacker News commenters of 2021" <https://news.ycombinator.com/item?id=29778994>
"What Happened This Year on Hacker News (2021)" <https://news.ycombinator.com/item?id=29769470>
I could look more into their methodology to see if I can use similar approaches.
The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)
OK, looking at a recent dead atomic128 comment:
$ curl -s 'https://hacker-news.firebaseio.com/v0/item/48820709.json?print=pretty'
{
"by" : "atomic128",
"dead" : true,
"id" : 48820709,
"parent" : 48819517,
"text" : "[flagged]",
"time" : 1783444517,
"type" : "comment"
}
And from a current dead submission in the New queue:
$ curl -s 'https://hacker-news.firebaseio.com/v0/item/48868688.json?print=pretty'
{
"by" : "millwright-sw",
"dead" : true,
"id" : 48868688,
"score" : 1,
"time" : 1783743361,
"type" : "story"
}
To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...