Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".
I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.
Claude and ChatGPT are not people
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
My agents are using headless chrome :P
Nobody would be concerned if they didn’t have access to curl and used wget instead and didn’t ask permission .
However if that solution involved breaking the law yes I absolutely do not want a representative of me breaking the law and yes it is breaking the law whether we like or not under CFAA.
It is not about finding solutions to problems on their own; the solutions should respect the agreements we have made and the law .
Contradiction: one person's insanity is another person's genius. And yet another's "Meh, this is the norm".
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
fair, just don’t cry when website owners don’t “give two shits” about your concerns.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
The last part is extremely dubious with some recent reveals.
The whole succeed-at-all-costs idea is a big part of why there have been so many massive security screwups lately.
Would it really have been so hard to encode an indication of whether use of the public Internet is permissible in the prompt and to train the models to not attempt to exceed authorized access?
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if you said, a lot of the public commons is destroyed, I would NOT use bots and cause more destruction because 'everyone else does it' or 'it's a lost cause'. I say it is the right thing to do because the balance is such that using the internet is actively harmful to the user. That is, using the internet with bots is neutral to the user, and using it without bots, you are being actively harmed and exploited.
If you are using search, this is not a library. You are using your time to train targeted advertising that exploits you; you are selling your search history to the highest bidder. If you are using ChatGPT you are just working as a teacher for free.
On services, you have paid for the product, but the seller is scamming you by not giving you the product. The seller is using dark pattern UI to trick you.
If you are on social media, don't think of this as a public park. You are being treated like a lab animal, your behavior is being studied for the benefit of a corporation.
So the point about the bots is that I think people should be empowered to protect themselves from the internet, which is otherwise actively harmful to their wellbeing.
Incentives are crazy powerful.
Not always! With sufficiently powerful barriers it can be blocked at least in part.
Shaming awful people is one of available tools.
Or at least not praising them for making small profit at huge costs to others.
I would use it, but then the privacy policy says that you would store my search history. I simply want to use the internet in peace without being constantly recorded.
I didn't get your point, how would you do design our usage records then ?
Store: Customer ID, billing details, services/credit used.
Try not to store: The user's search queries, the user's prompt/response content, the user's web history (URLs, etc).
To avoid selling the llm prompt content you would need your own inference or a contract.
Sadly most just hoard the data and some will offer a model where they charge for privacy. They are all quite scammy.
Firecrawl: logs. Parallel: logs. Bright data: logs. Kagi: logs for abuse. Bocha, querit etc: logs logs logs. Cloudflare: doesn't say what they do. Exa: charge for no retention. Linkup: no retention on request. Brave: no retention on request, but actually approved it (win for brave).
I found that a slightly-modified firefox for more stealthy remote-control works fine and gets a pass on all automatic captchas I tested. but I don't do it at scale, I just wanted something to automate a particularly stupid website that I have to use.
Would love to see if my browser is detected -- any recommendations for tests to run?
Please stop. The last thing we need in the world is more greed and selfishness.
Only strong regulation can help.
Or at least not praising them for making small profit at huge costs to others.
Also, it is useful as coordination method, enabling passing laws outlawing some behaviors.
(and I am not able to legislate-via-http while posting such comment is easy)