Why can't we simply save search stubs, short documents a LLM can retrieve, use them to route to various websites, resources, feeds, forums, blogs, other specialized search engines, APIs? If you host a 30GB model, then you can use 30 more GB for a compressed search stub index. The entry points for any domain are more stable than the pages. You would use an agent to navigate the resources as needed, fetching fresh data.
replyIt's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.
Actually that sounds quite logical
replyThat's a good idea. And if a team is working in a specific problem domain, their searches are likely to be repeated by others or overlap to a significant degree. With locally hosted agents that could probably be extended to caching the applicable search results (noting freshness, relevance, and accuracy) and have some sort of RAG-like internal search portal.
replyDiffbot might be the only company that offers a self-hostable fully airgapped web index with access to content cache (ala archive.org). I work here, happy to share more info about how it works!
replyY’all are honestly presenting your accuracy against a competitor right on your homepage - impressive.
replyLove this concept! Take the web “on a boat” :)