This is a problem of their own invention.

Nobody said you have to index the entire web.

The web would probably be a lot healthier if we had several small search engines that focused on niches rather than 5 failed search engines that tried to index everything that was ever written and then ended up paying Bing.