As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".

I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.

your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better