Overview
This page is the atomic definition. The deep-dive lives at crawl-budget.
Definition
Crawl budget is the number of URLs a search engine (most often Googlebot) is willing to fetch from a single site in a given window. Google defines it as the product of two factors: crawl capacity (how much load the server can take) and crawl demand (how much the search engine wants to recrawl). Sites under a few thousand URLs rarely hit the budget; large or fast-changing sites can starve important pages by spending budget on duplicates, faceted URLs, and infinite parameter combinations.
When it applies
Worry about crawl budget when the site exceeds roughly 10,000 URLs, ships large redirect chains, or relies on JavaScript rendering. Small static sites can ignore it.
Example
An ecommerce site with 50,000 products generates 2 million faceted URLs through filter combinations. Googlebot spends 80% of its budget on duplicate facets and misses new product pages. Adding robots.txt rules and rel="canonical" on facets consolidates the budget back to the canonical product URLs.
Related concepts
- crawl-budget - the deep-dive with diagnostic queries and fixes.
- technical - the broader technical SEO checklist.
- canonical-url - the canonical signal that consolidates crawl on one URL.
- indexnow - the push protocol that reduces wasted recrawl.
- internal-linking - link patterns that steer Googlebot to high-value pages.
Citing this term
See Crawl budget (llmbestpractices.com/glossary/crawl-budget).