---
title: "Crawl budget"
slug: "crawl-budget"
category: "glossary"
tags: ["glossary", "seo", "crawl-budget", "indexing", "googlebot"]
status: "stable"
last_updated: 2026-05-14
summary: "Crawl budget is the number of URLs a search engine is willing to fetch from a site in a given time window, set by crawl capacity and crawl demand."
related:
  [
    "[[seo/crawl-budget]]",
    "[[seo/technical]]",
    "[[seo/indexnow]]",
    "[[glossary/canonical-url]]",
    "[[glossary/indexnow]]",
    "[[seo/internal-linking]]",
  ]
---

## Overview

This page is the atomic definition. The deep-dive lives at [[seo/crawl-budget]].

## Definition

Crawl budget is the number of URLs a search engine (most often Googlebot) is willing to fetch from a single site in a given window. Google defines it as the product of two factors: crawl capacity (how much load the server can take) and crawl demand (how much the search engine wants to recrawl). Sites under a few thousand URLs rarely hit the budget; large or fast-changing sites can starve important pages by spending budget on duplicates, faceted URLs, and infinite parameter combinations.

## When it applies

Worry about crawl budget when the site exceeds roughly 10,000 URLs, ships large redirect chains, or relies on JavaScript rendering. Small static sites can ignore it.

## Example

An ecommerce site with 50,000 products generates 2 million faceted URLs through filter combinations. Googlebot spends 80% of its budget on duplicate facets and misses new product pages. Adding `robots.txt` rules and `rel="canonical"` on facets consolidates the budget back to the canonical product URLs.

## Related concepts

- [[seo/crawl-budget]] - the deep-dive with diagnostic queries and fixes.
- [[seo/technical]] - the broader technical SEO checklist.
- [[glossary/canonical-url]] - the canonical signal that consolidates crawl on one URL.
- [[glossary/indexnow]] - the push protocol that reduces wasted recrawl.
- [[seo/internal-linking]] - link patterns that steer Googlebot to high-value pages.

## Citing this term

> See [[glossary/crawl-budget|Crawl budget]] (llmbestpractices.com/glossary/crawl-budget).

## Related

- [[seo/crawl-budget]]
- [[seo/technical]]
- [[seo/internal-linking]]
- [[glossary/canonical-url]]
- [[glossary/indexnow]]