In this day and age, only good bots, the ones you actually want, obey robots.txt. You obtain no benefit by having a restrictive one.
replyThat is true, but if you have Anubis you presumably do intend to `Disallow` all bots.
replyOne of the standard library rules enables robots.txt passthrough by default. I can't control people not choosing to use it.
replyThat makes sense, Xe, but you don't enable it at:
reply- https://techaro.lol/robots.txt
- https://anubis.techaro.lol/robots.txt
and the latter (at least) is Anubis protected. It could be just an omission. But if it's intentional, I am just curious why so that as a bot author I can be well-behaved.
deleted
reply