LLM Best Practices

Markdown twins: serving page source to agents

3 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

A markdown twin is the raw markdown source of a page, served at a predictable URL so agents parse it instead of scraping rendered HTML. The llms.txt header announces the convention and a link tag in each page head makes it discoverable from any entry point. The index file itself is specified in llms-txt. Do not expect search benefit: Google’s AI optimization guide says Search needs no Markdown, and in February 2026 Google’s John Mueller and Bing’s Fabrice Canel both discouraged separate markdown pages built for LLM crawlers; Mueller questioned serving a page no user sees, and Canel warned of doubled crawl load.

Append .md to the page URL

Address the raw source by appending .md to the page URL, a convention that needs no lookup table. The llmstxt.org v2 spec accepts this and the extension-replacing form (page.html.md or page.md). An agent that must strip navigation and footers from HTML spends tokens on markup and can mangle code blocks; the source removes that step, and costs nothing at build time because the markdown already exists.

https://example.com/coding/python      -> rendered HTML, cite this to a human
https://example.com/coding/python.md   -> raw markdown, parse this

Announce the rule in the llms.txt header

State the rule in the llms.txt header, the first thing a crawler reads:

# Example Docs
 
> One-sentence description of the site.
 
Append `.md` to any URL below to get the raw markdown source instead of
rendered HTML. Every page is served both ways from this domain.

Add rel="alternate" in each page’s <head> so the markdown form is discoverable from any entry point. The llmstxt.org v2 spec defines this relation, plus rel="describedby" for the llms.txt that covers the page, as <link> tags or HTTP Link headers.

<link rel="alternate" type="text/markdown" href="/coding/python.md" title="Markdown source" />

Emit the tag only for URLs that have a twin. Generated routes such as tag listings and the 404 page usually have no source file, and advertising a URL that 404s is worse than advertising nothing. On a static site generator this is a build step that copies each source file to <slug>.md in the output, beside the emitter that already copies other assets.

Serve the same content to every client

Serve the twin at its own public URL to every visitor, with content identical to the HTML page. Do not switch the HTML response to an “AI-optimized” version by user agent. Bing’s Webmaster Guidelines prohibit cloaking, serving crawlers materially different content than visitors, and Canel and Mueller applied the same logic to bot-only markdown pages in February 2026.

Backlinks