Interesting. How much does it cost to build a search index in current year? Also, how do you get around the fact that everyone's blocking bots?
replyI've been thinking of making a "small web" indexer so I'm curious about that. I'm seeing even tiny websites being behind CloudFlare, Anubis etc. these days. (And everyone complaining they're getting hammered by mysterious distributed HTTP traffic!)
I have been working on an indexer for myself using data from common crawl. They do all the leg work and just host the outcome.
replyThat’s on me, didn’t see the text explaining it above the captcha.
replyI’ll give it a shot. Always happy to see some competition in the market. And I love the idea of handing out the source after a year.