Overview
Serve a markdown twin of every page so agents can parse the source instead of scraping rendered HTML. The twin lives at a predictable URL, the llms.txt header announces the convention, and a link tag in each page head makes it discoverable from any entry point. This page covers those three pieces; the index file itself is specified in llms-txt.
Append .md to the page URL for the markdown twin
Append .md to the page URL to address the raw source; it is the convention that needs no lookup table. An agent that has to strip navigation, sidebars, and footers out of HTML spends tokens on markup and sometimes mangles code blocks. Serving the source removes that step, and it costs nothing at build time because the markdown already exists.
https://example.com/coding/python → rendered HTML, cite this to a human
https://example.com/coding/python.md → raw markdown, parse this
Announce the rule in the llms.txt header
State the rule in the llms.txt header, not only on a separate agent page, because the header is the first thing a crawler reads:
# Example Docs
> One-sentence description of the site.
Append `.md` to any URL below to get the raw markdown source instead of
rendered HTML. Every page is served both ways from this domain.Advertise the twin with a link tag in the page head
Advertise it in the page <head> as well, so the markdown form is discoverable from any entry point rather than only from llms.txt:
<link rel="alternate" type="text/markdown" href="/coding/python.md" title="Markdown source" />Emit the tag only for URLs that actually have a markdown twin. Generated routes such as tag listings and the 404 page usually have no source file, and advertising a URL that 404s is worse than advertising nothing. On a static site generator this is a build step that copies each source file to <slug>.md in the output directory, alongside the emitter that already copies images and other non-markdown assets.
Serve the same content to every client
Serve the markdown twin at its own public URL to every visitor, and keep its content identical to the HTML page. Do not switch the HTML response to a different “AI-optimized” version based on the requesting user agent. Bing’s Webmaster Guidelines treat serving different AI-optimized versions to LLM crawlers as cloaking.