LLM Best Practices

How to run an SEO-safe site migration

4 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

A migration (domain change, HTTPS upgrade, URL restructure, or platform swap) rewrites the URL set Google has indexed. Make the redirect map the single source of truth, test it on a staging host, cut over in a quiet window, then watch real crawler traffic for gaps. Change one variable at a time; a simultaneous domain change, redesign, and CMS swap makes regressions impossible to attribute.

Prerequisites

  • Crawl access to the current site (Screaming Frog, Sitebulb, or wget --spider).
  • A staging host that can send X-Robots-Tag: noindex on every response.
  • Owner access to Google Search Console and Bing Webmaster Tools for the old and new properties.
  • DNS TTL lowered to 300 seconds at least 48 hours before cutover.
  • A baseline: 30 days of clicks, impressions, and average position per top URL.

Steps

1. Inventory every URL

Crawl and export every 200 URL with canonical, title, H1, and inbound links. Union it with the top URLs from Search Console (Performance, Pages, 16-month range) and the top referring URLs from a backlink tool. URLs that appear only in GSC or backlink data are orphans or off-site assets; redirect them too.

2. Build the 1:1 redirect map

One row per old URL, one new URL, no chains, no loops.

old_url,new_url,reason
https://old.example.com/blog/post-1,https://new.example.com/articles/post-1,renamed-folder
https://old.example.com/llms.txt,https://new.example.com/llms.txt,passthrough

Include robots.txt, sitemap.xml, llms.txt, ai.txt, and image and OG-card URLs. Missing the discoverability files is the most common mistake; see discoverability-files.

3. Stage on noindex and load the map

Serve staging with X-Robots-Tag: noindex, nofollow. A header also covers PDFs and images, which cannot carry a meta tag. Load the map into the platform’s redirect engine (Cloudflare Bulk Redirects, a _redirects file where the host supports one, or an Nginx return 301). Confirm with curl -I https://staging.example.com/.

4. Verify with a parallel crawl

Crawl staging and compare URL count, indexable count, and titles to the production crawl. Run every old URL through a checker:

while read u; do curl -s -o /dev/null -w "%{http_code} %{num_redirects} %{url_effective}\n" -L "$u"; done < old-urls.txt

Each must return 301 (or 308) and land on the mapped URL in one hop.

5. Flip DNS and remove the noindex header

Cut over in the hour with the lowest clicks over the last 30 days. Remove the noindex header the moment DNS flips, and confirm with curl -I https://example.com/ | grep -i x-robots-tag (no output is correct).

6. Submit and monitor for four weeks

Submit the new sitemap in GSC and Bing; for a domain change, use GSC’s Change of Address tool from the old property. Push changed URLs through indexnow (Bing and other participating engines; Google does not use it). Check daily for four weeks, then weekly:

  • GSC Page indexing: Not indexed counts trend down.
  • GSC Performance: clicks and impressions trend back toward baseline. Recovery time varies; four weeks is a house heuristic, not a Google timeline.
  • Server logs: any 404 with referrer traffic is a missing redirect; add it.
  • Redirect chains: re-run the checker weekly, since content changes add new ones.

Verify it worked

The top 50 old URLs each return one 301 to a 200, the sitemap shows Success in both webmaster tools, and no URL in the redirect map returns 404.

Common errors

  • Chains (/old to /mid to /new). They add latency and crawl cost, and Googlebot gives up after 10 hops; flatten them to one.
  • 302 instead of 301. A temporary redirect can keep the old URL indexed for a long time. Use 301 or 308; see redirects.
  • Missing image, OG, robots.txt, sitemap.xml, or llms.txt redirects.
  • The staging noindex header still set after cutover. The new site stays invisible until it is removed.

Backlinks