Does it work though? The big LLM crawlers do not read llms.txt so will they read and follow the same instructions as HTML?
replySomeone has to have done or is doing an experiment with this right? I also think that if it was an actual profitable thing then we would know about it pretty quickly. It would pop up everywhere
replyApparently new checks in Chrome Lighthouse are checking for the existence of the file.
replyhttps://searchengineland.com/google-llms-txt-chrome-lighthou...
Yeah I want to know how many donations they get
replyI have no idea, in theory it might catch some miss-configured agents off-guard
replyWhere did you see the big crawlers don't read it? Anthropic does.. they're pretty big.
reply