This is not meaningfully static content. Look at the charts in TFA. There is a combinatorial explosion of distinct URLs which the crawlers can and do request.
replyThey allow you to diff commits, which is an awesome feature.
replyBut if bots a crawling diffs between all possible commits it's crazy.
CDN will do nothing, because it's new urls each time. You can maybe find a CDN provider that block bots.
No CDN blocks bots. Some claim to. None do.
replyDid you read the article?
reply“[…]because we can generate 1.2 METRIC BAJILLION valid URLs just for a single fork of linux.git.”
From the HN guidelines <https://news.ycombinator.com/newsguidelines.html>:
reply> Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that".
What if it's really obvious they didn't read the article?
reply