Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
[0]: https://github.com/EratoLab/erato
[1]: https://github.com/EratoLab/web-search-mcp
Obviously, hosting a search index locally would be best practice, for the search part, but usually comes with quite a significant hardware cost and operational complexity, if you want to have an up-to-date index. I'm not aware of good self-hostable options in the space but would be happy to hear about them if anyone here has experience with that! AFAIK LinkUp and some other provides do offer private cloud/on-premise deployments, but I think mostly for their biggest customers.
For the web fetch/access part of the equation, that is inherently live and not containable for a local deployment (unless you would host a whole internet archive/Wayback Machine, which is even more of an undertaking). There, the main options you have to prevent leaks, are a domain allowlist (if feasible), doing query parameter stripping (error prone, and still no 100% guarantee), or configuring the client (e.g. our tool) in a way that users have to explicitly allow those tool calls, and are informed about the posed risks. We are also working on a system, where certain tools are fully deactivated once e.g. an office document that's marked as confidential enters the context.
For the highest requirements on company secrets, an air-gapped system that doesn't have web search/access capabilities is obviously the ultimate solution.
It's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.
You claim VISA and other credit card companies don't mind if merchants like DeepSeek steal hundreds of thousands of credit cards and use them? And still continue to give them merchant accounts to continue with that?
Isn't it more likely that some people are scaremongering?
> And still continue to give them merchant accounts to continue with that?
The AI companies don't steal the credit cards. They instead fund unaffiliated grey-market transfer stations that steal credit cards. They feign compliance by simply funding another transfer station.
"Anthropic/OpenAI bad!" cynicism is not a replacement for critical thinking (as it increasingly seems to be on HN).
I am so glad we never see stories against Anthropic/OpenAI claiming that! Thankfully we don't even have to search, critical thinking is enough to know that Anthropic/OpenAI are responsible data sovereign and others are not.
Anyways, they can easily be sued by DeepSeek if they are lying.
> critical thinking is enough to know that Anthropic/OpenAI are responsible data sovereign and others are not
Critical thinking is not relying on a confirmed bad actor because you think the others maybe could be bad actors.
For others, here's your blog post:
https://www.anthropic.com/threat-intelligence-report-septemb...
Do we? Anthropic made allegations, but have they substantiated them?
DeepSeek could sue for defamation if they are lying. There's a reason they aren't.
Credit card fraud and the illegal data routing is a crime, but that's something the affected customers would need to sue for.
There is a difference between "white-knighting" and pointing out that maybe businesses shouldn't use an API that has been proven to illegally route data elsewhere, resulting in literal state secrets being leaked.
There is nothing wrong with using DeepSeek's models locally or on your own cloud, and using another search provider! But using DeepSeek's own APIs is pure idiocy unless you really don't care about your data security/privacy (and that of your users).
If you're asking about my biases: I am significantly invested in AI (all US labs including Anthropic/OpenAI, and the hardware supply chain) because I believe it will be the most transformative tech in human history. I am also American, and so I'd prefer not to have a nation that blends thousands of human rights activists getting to superintelligence first.
My investments in the frontier have grown so I do not claim to be unbiased.
Another bias - I think HN's user base is on average consistently shortsighted when it comes to this technology:
1/ The HN zeitgeist has been consistently wrong about AI since LLMs were introduced. Old HN threads are hilarious time capsules demonstrating how headstrong+dumb the median HN opinion is. These same people post today as if they never said anything dumb a year ago, in the same overconfident and cynical tone.
2/ The median technologist's takes here are actively destructive to the likelihood of us surviving superintelligence. As technologists we all have a responsibility to ensure this goes well - our companies and societies look to our opinions. If people here introspected a bit more and reflected on the previous times they were wrong, and saw this tech for what it really is, we would all have a better chance of surviving superintelligence.
Has your astroturfing manual not changed over the past few months?
I don't think that X post from someone that sells relays is meaningful. Obviously Anthropic is aware of what relays are, they aren't dumb enough to conflate relays with DeepSeek.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
I’m not surprised. Google these days regularly blocks me, a human user with a Google account with decades of tenure.
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
https://arxiv.org/abs/2404.01019
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
Your understanding of the world "garden" probably came from 5+ encounters with the word in different contexts, and you cannot separate them - instead your understanding is some kind of average of the knowledge from each encounter.
Interestingly, adjacent and overlapping-meaning words like park, meadow, yard, grassland etc often have disputed boundaries - ie. One person might call something a garden, whilst someone else calls it a meadow. Neither can explain quite why they have chosen their specific word for the object in question, but it might stem from the source of where the meaning of each word was initially learned.
The Saussure idea of the differential nature of language seems relevant here too: words aren't just lists of things you look up, they are connected to each other and distinguish themselves by being slightly different from the other elements of the clusters they belong to.
It also highlight that Google really isn’t about search, all they care about is advertising and making users profiles.
A paid search API seems like it would be easy money for Google. And maybe some lock-in for all sorts of start-ups and enterprises.
There are so many apps that would benefit from search.
I get that it means they can't serve ads.
But you've got to be willing to disrupt yourself or you'll lose eventually anyway.
Better to figure out how to make money from ads and an API, than to cower in fear.
That same fear may be why they failed to be first movers in LLMs.
Have to be willing to disrupt yourself.
At the same time, I've never built a billion dollar company. So I'm absolutely armchair quarterbacking here.
We should have outlawed targeted advertising and Google should have been broken up under antitrust a long time ago.
Easy enough to run the ad-engine over the query, estimate the ad revenue, and bill the user the value of the lost revenue.
They are free to recoup that by serving their own ads or even using Google's ad services.
Exactly why it seems to make sense to be willing to disrupt yourself.
There are so many ways to still make money.
But, I'd be willing to bet that avoiding self-disruption increases the likelihood of failure over longer timescales.
But it does seem like their index isn’t great.
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
Source: https://uruky.com/faq
API: https://uruky.com/docs/api
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
1. Tavily Basic: 1,704 searches/month
2. Brave Search: 2,126 searches/month
3. Exa Fast / Auto: 2,233 searches/month
4. Exa Instant: 3,907 searches/month
5. DeepSeek Search: 7,503 searches/month
6. Ceramic: 22,507 searches/month
The free tiers on most are generous enough for normal agent MCP usage
Also...pretty sure decodo just launched a search thingie
I’m not accusing any one engine discussed in this thread, but hearing a proxy vender like decodo providing search give me a feeling there’s non zero over lap to bad search and residential proxy crawler.
I build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.
You can't really get anywhere these days without disguising UA etc.
Anyway I’m not expecting a solution. Just say this sux.
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
https://artificialanalysis.ai/agents/search-api?search-api-c...
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".
I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.
Claude and ChatGPT are not people
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
My agents are using headless chrome :P
Nobody would be concerned if they didn’t have access to curl and used wget instead and didn’t ask permission .
However if that solution involved breaking the law yes I absolutely do not want a representative of me breaking the law and yes it is breaking the law whether we like or not under CFAA.
It is not about finding solutions to problems on their own; the solutions should respect the agreements we have made and the law .
Contradiction: one person's insanity is another person's genius. And yet another's "Meh, this is the norm".
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
fair, just don’t cry when website owners don’t “give two shits” about your concerns.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
The last part is extremely dubious with some recent reveals.
The whole succeed-at-all-costs idea is a big part of why there have been so many massive security screwups lately.
Would it really have been so hard to encode an indication of whether use of the public Internet is permissible in the prompt and to train the models to not attempt to exceed authorized access?
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if you said, a lot of the public commons is destroyed, I would NOT use bots and cause more destruction because 'everyone else does it' or 'it's a lost cause'. I say it is the right thing to do because the balance is such that using the internet is actively harmful to the user. That is, using the internet with bots is neutral to the user, and using it without bots, you are being actively harmed and exploited.
If you are using search, this is not a library. You are using your time to train targeted advertising that exploits you; you are selling your search history to the highest bidder. If you are using ChatGPT you are just working as a teacher for free.
On services, you have paid for the product, but the seller is scamming you by not giving you the product. The seller is using dark pattern UI to trick you.
If you are on social media, don't think of this as a public park. You are being treated like a lab animal, your behavior is being studied for the benefit of a corporation.
So the point about the bots is that I think people should be empowered to protect themselves from the internet, which is otherwise actively harmful to their wellbeing.
Incentives are crazy powerful.
Not always! With sufficiently powerful barriers it can be blocked at least in part.
Shaming awful people is one of available tools.
Or at least not praising them for making small profit at huge costs to others.
I would use it, but then the privacy policy says that you would store my search history. I simply want to use the internet in peace without being constantly recorded.
I didn't get your point, how would you do design our usage records then ?
Store: Customer ID, billing details, services/credit used.
Try not to store: The user's search queries, the user's prompt/response content, the user's web history (URLs, etc).
To avoid selling the llm prompt content you would need your own inference or a contract.
Sadly most just hoard the data and some will offer a model where they charge for privacy. They are all quite scammy.
Firecrawl: logs. Parallel: logs. Bright data: logs. Kagi: logs for abuse. Bocha, querit etc: logs logs logs. Cloudflare: doesn't say what they do. Exa: charge for no retention. Linkup: no retention on request. Brave: no retention on request, but actually approved it (win for brave).
I found that a slightly-modified firefox for more stealthy remote-control works fine and gets a pass on all automatic captchas I tested. but I don't do it at scale, I just wanted something to automate a particularly stupid website that I have to use.
Would love to see if my browser is detected -- any recommendations for tests to run?
Please stop. The last thing we need in the world is more greed and selfishness.
Only strong regulation can help.
Or at least not praising them for making small profit at huge costs to others.
Also, it is useful as coordination method, enabling passing laws outlawing some behaviors.
(and I am not able to legislate-via-http while posting such comment is easy)
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
I'm kind of surprised nobody has done that yet actually. I guess Opus 5.5 has only been out for a very short time...
It wouldn't be for free though, would it? People need to build and maintain it (and there's LLM cost associated with it too).
Let's say you are a 5 person start-up. It makes much more sense to pay $15/user/month for services like Jira, Slack, Email etc. then to build and maintain everything yourself.
For super large companies, it might make financial sense to build some of the tools in-house. But then again, that probably already made sense for LLMs were a thing. But there is more to it than just the pure financials (i.e. salary).
Maybe a 5 person company wouldn't do it. A 100 person one though? No brainer. And it only takes one to open source their solution.
Slop curators and tech support shoulders to cry on. Could be lucrative.
There are plenty of use cases for the ability to quickly through together and refine an application without reviewing the code though.
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
"People said it before and it wasn't true then" isn't really an argument about it being true now. There are plenty of things that happened where the zealots jumped the gun with the hype. Bitcoin for example. "People said bitcoin was going to succeed in 2005, 2006, 2007...". Or even the web. It was mega hyped in the dot com boom and then crashed and of course loaaads of stupid people thought that was the end of it.
the moment there is a model which can code whole app, we can fairly say software engineering is done.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
You are right that there's plenty of use cases where Opus 5.5 is better.
Not everyone wants Anthropic to succeed.
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
That is explicitly the Chinese strategy.
Now, what if the largest capitalist economy went all-in on an AI that might no longer be sufficiently monetizable?
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
> First, we should adhere to the principle of openness and win-win and boost innovation-driven development. As a new engine of world economic growth and an accelerator for the shift of growth drivers, AI is moving from the digital world into the physical world. We should seize this rare, historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries and forward-looking planning for future industries, so that all sectors and businesses can benefit from AI
-- Keynote speech by Chinese President Xi Jinping at opening ceremony of 2026 World AI Conference
http://english.scio.gov.cn/topnews/2026-07/18/content_118605...
As a socialist myself, the above comment looks like a prime example of what Mark Fisher would call capitalist realism, the ideological belief that there is no alternative. The fact that somebody would think past quarterly earnings appears unthinkable to the committed ideologue.
They wouldn't give away R&D for free, they would give it away to destroy the capitalist mode of production, as outlined above. If that isn't a worthy goal from China's perspective, then I don't know what would be.
China isn't the US. They don't share the same ideology, strategy or conclusions.
As for the export controls, they explicitly exempt already-open models and set up an automated streamlined process for future ones.
Well... obviously. What the fuck is the point of export controlling an already open model.
> for future ones
Nope. >2T models (i.e todays frontier) is explicitly not allowed.
Also see: https://www.reuters.com/world/asia-pacific/china-proposes-ad...
> China isn't the US. They don't share the same ideology, strategy or conclusions.
Money and power know no ideology. They do pretty much the exact same things as the US as well (from a 3rd country's POV). They expand territory, they capture industries, etc, etc, etc,.
> Keynote speech by Chinese President Xi Jinping
Do you take the same literal interpretations of other politicians' speeches? Surely you know thats politicking, the same way every G2/G7 leader says they want a collaborative world blah blah but it's naive to not know that it is cutthroat competition in reality.
> they would give it away to destroy the capitalist mode of production
Well, I won't take that dream away from you, but you're gonna be disappointed in this particular case, in our lifetime, with this particular country, atleast :)
If Anthropic and OpenAI economics are as bad as they seem to be they deserve to fail, others will take over the remaining market. It’s not a team sport
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
Anthropic is most likely not. Despite bilions of state funding and contracts.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
I understand if you pay is a different story.
I mean ... "how does this work?"
They're entering the search engine scene since they already have "all of the internet" stored due to the expansive LLM crawling -> training?
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
egress on fetch looks to be huawei cloud in hong kong
fetch tool uses a linux firefox 149 user agent with lang set to china. it runs js. could be using something like camoufox.
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.
Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.
That is a very respectable motto for any company.
If you are talking about Gemini ( and other AI providers) no they don't they simply steel content.
I am asking to know if DS is any different because I failed to find this information on their website.