You might want to bookmark https://openwebsearch.eu/open-webindex/
replyWhile the index is currently not open source, it should be at some point. Maybe when they get out of the beta stage (?) details are yet unclear.
You know, it's possible the cure to an adversarial internet is to just have some non-profit serve as a repo for a universal clearnet index that anyone can access to build their own search engine. That way we don't have endless captchas and anubis and Cloudflare tests every time I try and look for a recipe online. Why send AI scrapers to crawl literally everything when you're getting the data for free?
replyI'll add it to the mile-long list of things that should exist and be online public goods.
Is the common crawl usable for something like this?
replyI'm the creator of searcha.page and seek.ninja, those are the basis of my index. The biggest problem with ONLY using that is freshness. I've started my own crawling too, but for sure common crawl will backfill a TON of good pages. It's priceless and I would say common crawl should be any search engines starting point. I have 2 billion pages from common crawl! There were a lot more but I had to scrub them out due to resources. My native crawling is much more targeted and I'd be lucky to pull 100k but as long as my heuristics for choosing the right targets it will be very high value pulls.
replyMost likely it is, the issue then becomes being able to store and afford the storage for all the files.
replySure, and that's not easy, but it's a lot easier than having to crawl the entire public Internet yourself.
replyWhy can't crawling be crowd sourced? It would solve ip rotation and spread the load
replyThat’s how residential proxies work, in a perverse way
replyCommon crawl sort of serves this function. I use it. It's a really good foundation.
replyreply
Too bad it doesn't support android. It is much more energy efficient than anything else I can spare (for 100% uptime contribution)
replydeleted
replyThe IP thing is interesting, I was trying to make this CSGO bot one time to scrape steam's prices and there are proxy services out there you rent, tried at least one and it was blocked by steam. So I wonder if people buy real IPs.
replyYeah people buy residential IPs on the black market. They are essentially infected home PCs and botnets.
replyNot just the black market anymore!
replyyou can get paid about $0.10/GB in cryptocurrency (at a few GB per month) to run one on your PC. Apparently they also just buy actual connections sometimes. It's not even unethical - it's just two groups of equally bad businesspeople trying to spend money to block the other one.
replyThe crawl seems hard but the difference between having something and not having it is is very obvious. Ordering the results is not. What should go on page 200 and do those results still count as having them?
reply