The crawlers would have still just hammered their site though, right?
replyI think the idea is that they could store the data in BigQuery, and point users of the site there.
replySure, but now you're moving the site from "Free data presented in a pleasant way to view" to a "pay-as-you-go database". Your audience shifts dramatically, and you lose the ability to share the data you're trying to present.
replyTrue. But it sounds like they already lost that.
replyThe crawlers would have still just hammered their site though, right?
replyWhy is your comment exactly word for word of another comment just one level above in the comment chain?
replyMaybe because they restated what they said instead of addressing to the previous commenter’s point.
replyBecause the crawlers would still have hammered their site, though.
reply(The GP post doesn’t actually meaningfully address the issue being raised. Adding BigQuery or whatever would not change the fact that (a) they already offered a method of getting all of the data in a cost effective way, and (b) the issue was that the crawlers hammered the site hard enough to make it economically unviable.)
I think snowflake has similar, you can rent it out or make it free
reply