LLM Best Practices

Crawl Budget

4 min read

View markdown source

Overview

Crawl budget is the number of URLs Googlebot will fetch from a site within a window. Google’s own crawl budget guide says most sites do not need to worry about it; new pages on smaller sites are usually crawled the same day they are published. On large or fast-changing sites, crawl budget can become the bottleneck between publish and rank. Block low-value paths, consolidate duplicates, and signal freshness deliberately.

Know when crawl budget matters

Google’s guide names the cases where crawl budget is worth managing.

  • Large sites (around 1 million or more unique pages) whose content changes moderately often, such as weekly.
  • Medium or larger sites (around 10,000 or more unique pages) whose content changes very rapidly, such as daily.
  • Sites with a large share of URLs reported as “Discovered, currently not indexed” in Search Console.
  • In practice, add e-commerce, classifieds, and UGC sites with parameter explosion.

Below those thresholds, Google’s discovery scheduler covers the site comfortably. Spending engineering time on crawl budget for a 200-page site is wasted; spend it on content and internal linking. See internal-linking for the depth rules that also affect crawl rate.

Block low-value paths in robots.txt

robots.txt is the first line of crawl budget defense. Block anything that produces URLs but not rankings.

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /search?
Disallow: /*?sort=
Disallow: /*?filter=
Disallow: /cart
Disallow: /account/

Sitemap: https://example.com/sitemap.xml
  • Faceted navigation with infinite filter combinations: block.
  • Internal search result pages: block.
  • Session-id URLs, cart pages, user account pages: block.
  • Test domains and staging: block at the User-agent: * level, not just the staging robots.txt.

Disallow prevents crawling but does not prevent indexing if the URL is linked from elsewhere. Pair with noindex for URLs that exist but should not appear in search.

Apply noindex to thin and duplicate pages

noindex,follow removes the URL from the index while letting crawlers traverse it. Use it on pages that exist for users but not for search.

  • Thank-you pages and confirmation flows.
  • Author archive pages with only one or two posts.
  • Tag archives that duplicate category archives.
  • Infinite-scroll paginated archives past page 5.
  • Internal-only documentation.

noindex is a meta tag, not a robots.txt rule. Robots.txt-blocked URLs cannot receive noindex because the crawler never fetches the meta tag.

<meta name="robots" content="noindex,follow" />

Use canonical tags to consolidate duplicate URLs

When the same content lives at multiple canonical URLs (sort orders, tracking parameters, pagination), canonical to one URL.

  • Every page declares a self-referential canonical. Parameter variants point back to the clean canonical. See technical.
  • ?utm_source=... URLs canonical to the unparameterized URL.
  • Sort and filter parameters on category pages canonical to the unsorted version.
  • Mobile and desktop variants canonical to the responsive single URL. See mobile-first.

Canonical is a hint, not a directive. Google may ignore the canonical if the pages diverge significantly; 301 redirects are stronger when the URLs are truly equivalent.

Signal freshness through accurate sitemap <lastmod>

Google uses <lastmod> to schedule recrawls when the value is consistently accurate, so set it only when content materially changes, never to the build time. Pair the sitemap with indexnow for push notification to Bing and other participating engines. See sitemaps-deep for the full <lastmod>, changefreq, and priority rules.

Self-canonical every paginated page

Point each paginated URL’s canonical at itself, not at page 1, and expose real ?page=N URLs behind infinite scroll so crawlers can walk the list without running scroll logic. Google stopped using rel="next" and rel="prev" in 2019. See pagination-and-facets for the pagination and facet rules in full.

Monitor crawl behavior in Search Console

Google Search Console’s Crawl Stats report (Settings > Crawl stats) is the only ground truth.

  • Total crawl requests per day: watch the trend against how much you publish and update.
  • Crawl response codes: most requests should return 200. Google reduces its crawl rate when it sees server errors (5xx) or connection problems; a spike needs fixing.
  • Response time: Google raises crawl capacity when the server responds quickly and lowers it when responses slow down. Google publishes no fixed millisecond threshold.
  • File type breakdown: a high ratio of image and CSS crawls vs HTML crawls means the crawler is wasting budget on assets.

Consolidate duplicate content before crawl-budget tactics

The cheapest crawl-budget win is shipping fewer URLs. Audit for duplicates first.

  • Tag pages duplicating category pages: remove one set, 301 to the other.
  • Multiple URLs serving the same content: pick one canonical, 301 the rest.
  • Soft-404s (pages that return 200 with “no results found”): return real 404s.
  • Thin pages with no real intent: consolidate or delete. See the thin-content rules in content.

Backlinks