upvote
Can you explain how this would help protect a company's secrets? I would think that if you want to keep your IP secret surely using something like Deepseek's API directly is not ideal and hosting something locally is best practice.
reply
I personally also wouldn't trust Deepseek with companies secrets. That's why I linked LinkUp, which I know has passed through a quite strict compliance process at a few of our customers.

Obviously, hosting a search index locally would be best practice, for the search part, but usually comes with quite a significant hardware cost and operational complexity, if you want to have an up-to-date index. I'm not aware of good self-hostable options in the space but would be happy to hear about them if anyone here has experience with that! AFAIK LinkUp and some other provides do offer private cloud/on-premise deployments, but I think mostly for their biggest customers.

For the web fetch/access part of the equation, that is inherently live and not containable for a local deployment (unless you would host a whole internet archive/Wayback Machine, which is even more of an undertaking). There, the main options you have to prevent leaks, are a domain allowlist (if feasible), doing query parameter stripping (error prone, and still no 100% guarantee), or configuring the client (e.g. our tool) in a way that users have to explicitly allow those tool calls, and are informed about the posed risks. We are also working on a system, where certain tools are fully deactivated once e.g. an office document that's marked as confidential enters the context.

For the highest requirements on company secrets, an air-gapped system that doesn't have web search/access capabilities is obviously the ultimate solution.

reply
Why can't we simply save search stubs, short documents a LLM can retrieve, use them to route to various websites, resources, feeds, forums, blogs, other specialized search engines, APIs? If you host a 30GB model, then you can use 30 more GB for a compressed search stub index. The entry points for any domain are more stable than the pages. You would use an agent to navigate the resources as needed, fetching fresh data.

It's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.

reply
Actually that sounds quite logical
reply
That's a good idea. And if a team is working in a specific problem domain, their searches are likely to be repeated by others or overlap to a significant degree. With locally hosted agents that could probably be extended to caching the applicable search results (noting freshness, relevance, and accuracy) and have some sort of RAG-like internal search portal.
reply
Diffbot might be the only company that offers a self-hostable fully airgapped web index with access to content cache (ala archive.org). I work here, happy to share more info about how it works!
reply
Y’all are honestly presenting your accuracy against a competitor right on your homepage - impressive.

Love this concept! Take the web “on a boat” :)

reply
I would use this to discover leaks, not to protect data.
reply