Hacker News
new
past
comments
ask
show
jobs
points
by
hahnbee
6 hours ago
|
comments
by
charles_f
6 hours ago
|
[-]
Not op, but a common tactic is to put honeypots in a page set to be forbidden in Robots.txt. Not respecting Robots.txt is making you a bad crawler. I have marginal blog, but its somewhat indexed and it gets tons of traffics from crawlers
reply