Google has recently added `llms.txt` to Chrome's Lighthouse check for agentic browsing (https://searchengineland.com/google-llms-txt-chrome-lighthou...), so adoption may be coming. Admittedly, I put more faith in
reply <link rel="alternate" type="text/markdown" href="https://example.com/foo.md" title="Markdown version of the <Foo> page">
I have created an `llms.txt` for my website anyhow. I use a fixed LLM prompt to generate it from the internal links in `index.md`.
Giving a markdown version of a page seems like an interesting choice instead of just embedding a schema marked up one
replyEvery page on code.claude.com has a markdown version available by just appending ".md", and Claude Code knows about it. E.g:
replyAfter some consideration, I also applied this convention to every site I build - including content negotiation: Clients can either send an Accept header with their preference, or append an explicit extension (.md|.markdown for Markdown, .json for JSON API responses, or .html for the human HTML page). Together with the content negotiation part, it feels very much like HTTP was intended to work - especially the fact that API clients, AI agents, and humans all use the same URLs, but get the content in the shape they need.
replyI've done this off and on for various sites over the years too, and probably should be more consistent about it. A number of sites do or used to do some variation of this, and I wish it was more widespread. E.g. Reddit will serve up a json version of a sub-dreddit if you do /r/subreddit.json
replyHere's how to do it with more recent versions of Hugo:
replyhttps://photostructure.com/coding/hugo-markdown-output/
(It includes the grandparent's head link suggestion, but it's not just "change .html to .md" because I'm old skool and as a wee nerd was told that URLs ending in .html or .php or whatever we're frowned on, so the above link's markdown is available by appending /index.md )
deleted
replyIt gets even more "interesting" for markdown-based systems like Astro or Obsidian Publish: author in md -> ship html && optionally serve md?
replyThe same could be said of robots.txt
replyAnd anything else that might tell them not to access something.
robots.txt predates the modern web though
replyMy point was that llms.txt not working is no different from them ignoring everything else that came before and probably everything that is yet to come.
replyIf they want it, they will take it, polite directives in text files will have no effect.
To be fair, "not adopted by any major AI player" is probably the most web-standard-compliant phase of a new web standard.
reply