Overview
A public site ships about a dozen small files at fixed paths that crawlers, agents, browsers, and security researchers look for without negotiation. This page is the master catalog: each file, its location, its purpose, body rules, and the deep-dive link. Use it as the pre-launch checklist and the post-launch audit list.
Know which files search engines use. Google Search and Bing rely on robots.txt and sitemaps, and Bing also on IndexNow. llms.txt, agents.txt, ai.txt, and /llm-info are house-standard items for agents and tools: Google’s AI optimization guide says Google Search ignores llms.txt and that its AI features need no special files.
The catalog
| File | Path | Purpose | Required if |
|---|---|---|---|
robots.txt | /robots.txt | Crawl scope and sitemap pointer | Always |
sitemap.xml | /sitemap.xml | Every canonical URL with lastmod | Always |
llms.txt | /llms.txt | Agent-facing index of canonical pages | Always (house standard core item) |
llms-full.txt | /llms-full.txt | Full text of priority pages for agent ingestion | Small vault sites and reference docs |
agents.txt | /agents.txt | Agent policy and start-here pointer | Always (house standard core item) |
/llm-info page | /llm-info | Machine-readable site description | Always (house standard core item) |
ai.txt | /ai.txt | AI usage policy declaration | Always (house standard, permissive) |
| IndexNow key | /<key>.txt | Ownership proof for IndexNow pings | Using IndexNow for Bing or Yandex |
security.txt | /.well-known/security.txt | Vulnerability disclosure contact (RFC 9116) | Production sites |
humans.txt | /humans.txt | Credit for the people behind the site | Optional |
| Web manifest | /manifest.webmanifest or /manifest.json | Install metadata, icons, theme | Mobile traffic matters |
favicon.ico | /favicon.ico | Browser tab and SERP icon | Always |
| OG default image | /og-default.png (1200x630) | Fallback social card | Always |
| Apple touch icon | /apple-touch-icon.png (180x180) | iOS home-screen icon | iOS traffic matters |
Rules per file
robots.txt
Minimum body, one Sitemap: line per sitemap, and Disallow: only for paths that should not be crawled. Add comments pointing at /llms.txt and /agents.txt. Deep dive: technical.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml
sitemap.xml
List every canonical URL with a real <lastmod> taken from the page’s last_updated, regenerated at build and never hand-edited. Split into a sitemap index past 50,000 URLs. Submit once in Google Search Console and Bing Webmaster Tools; engines refetch on their own. Pair with indexnow: Bing’s July 2025 guidance recommends sitemaps plus IndexNow for AI-powered search.
llms.txt and llms-full.txt
/llms.txt is the agent-facing markdown index, served as text/plain or text/markdown; llms-txt has the format and ship-llms-txt the build steps. /llms-full.txt concatenates the priority pages for one-fetch ingestion; keep it within a model context and generate both from the same frontmatter.
agents.txt, /llm-info, ai.txt
agents.txt is the key-value agent policy (agents-txt). /llm-info is a crawlable HTML page with a machine block and an “INSTRUCTIONS FOR AI ASSISTANTS” section, linked from the sitewide “Hey AI, learn about us” footer link; spec in llm-discoverability-standard. ai.txt declares AI-usage preferences, permissive under the house standard (ai-txt).
IndexNow key file
A flat text file at /<key>.txt whose body is the same key, 8 to 128 characters of letters, digits, and dashes. Bing and Yandex require it before accepting pings. Deep dive: indexnow.
security.txt
Place it at /.well-known/security.txt, served over HTTPS as text/plain. RFC 9116 requires Contact: (a mailto: or https: URI) and Expires: (ISO 8601), and recommends an expiry under a year out. Put the renewal on a calendar, because an expired file tells researchers the contact may be dead, and route Contact: to a monitored inbox rather than a personal account. Optional fields: Preferred-Languages:, Canonical:, Policy:, Acknowledgments:, Encryption:, Hiring:.
Contact: mailto:security@example.com
Expires: 2027-01-01T00:00:00Z
Preferred-Languages: en
Canonical: https://example.com/.well-known/security.txt
humans.txt
humans.txt credits the team, tools, and stack. No engine reads it and no standard governs it; the humanstxt.org convention uses headers like /* TEAM */. Ship it for credit or skip it.
Web manifest, favicon, Apple touch icon, OG default
Reference the manifest from every page head and include at least a 192x192 and a 512x512 PNG in icons, plus name, short_name, start_url, display, theme_color, and background_color. Validate in the Chrome DevTools Application panel.
<link rel="manifest" href="/manifest.webmanifest">
<link rel="icon" href="/favicon.ico">
<link rel="apple-touch-icon" sizes="180x180" href="/apple-touch-icon.png">
<meta property="og:image" content="https://example.com/og-default.png">Keep /favicon.ico at the root even when an SVG icon is declared, since some clients request it directly. Use an absolute URL for the default OG image; per-page images override it (og-images). The full icon tag set is in html-head-tags.
Pre-launch checklist
Run once before flipping DNS to production; every line should return 200.
HOST="https://example.com"
for path in /robots.txt /sitemap.xml /llms.txt /llms-full.txt /agents.txt \
/ai.txt /llm-info /.well-known/security.txt /humans.txt \
/manifest.webmanifest /favicon.ico /og-default.png \
/apple-touch-icon.png; do
status=$(curl -s -o /dev/null -w "%{http_code}" "$HOST$path")
echo "$status $path"
doneThese paths only count at the origin root; a project-subpath deploy (user.github.io/project/) serves them where no crawler looks. Build wiring for a Quartz site: static-site-seo.