LLM Best Practices

Pagination and faceted navigation

4 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Pagination splits long lists across numbered URLs; faceted navigation lets users filter them by attributes such as color, size, or price. Both create URLs a crawler must decide about. Handled well, every product or post page gets indexed; handled badly, crawl budget burns on filter combinations while long-tail listings stay unindexed.

Make every paginated page self-canonical

Each numbered page declares its own URL as canonical. Google says “don’t use the first page of a paginated sequence as the canonical page,” and it no longer uses rel="next" and rel="prev" (since 2019), so those tags are inert for Google.

<!-- on /blog/page/3 -->
<link rel="canonical" href="https://example.com/blog/page/3" />

Canonicalizing pages 2 to N to page 1 is the most common error. It looks tidy but removes every item on those pages as a ranking target.

  • Give each page a distinct URL (?page=3 or /page/3). Google ignores # fragments, so a fragment-only difference may not be followed.
  • Link pages with real <a href> elements.

Keep deep pages reachable in two or three clicks

A post on page 47 of a flat list is 47 clicks deep. Fix it with numbered pagination that always links the first page, the last page, and the current page’s neighbors, or with hub pages by date, tag, or category (/blog/2025/ linking every post from that year). See site-architecture and internal-linking.

Disallow crawl-wasting facet patterns in robots.txt

Facets multiply URLs (/shoes?color=red&size=10), and most combinations are slices of the parent category. Google’s faceted-navigation guidance ranks the options: disallow facet URL patterns in robots.txt first, because crawling filtered items rarely pays off, then URL fragments, rel="canonical" to the unfiltered page (slower and weaker), and rel="nofollow" on every internal link to the filter URL.

User-agent: *
Disallow: /*?sort=
Disallow: /*?price=
Disallow: /*&color=
  • Use this for parameters that never make a useful landing page.
  • A disallowed URL cannot show a canonical or noindex, and it stays blocked until you remove the rule.
  • For facet URLs you do want crawled, separate parameters with & (commas, semicolons, and brackets are hard for crawlers to read as separators), keep path-encoded filters in a fixed order with no duplicates, and return 404 for combinations that have no results, including nonsensical ones. Do not redirect them to a generic error page.

Use noindex only where crawling is cheap

noindex,follow keeps a crawlable filter URL out of the index and lets crawlers walk its product links, but Google still fetches the page to read the tag, so it does not save crawl budget. It suits small catalogs. Point the canonical at the unfiltered category and promote only the few facets with proven query demand (a brand, a top category) to indexable landing pages with their own content. Never put Disallow and noindex on the same URL.

<!-- on /shoes?color=red&size=10 -->
<meta name="robots" content="noindex,follow" />
<link rel="canonical" href="https://example.com/shoes" />

Render real URLs behind infinite scroll

Google’s crawlers do not click buttons or trigger user-action scripts, so infinite scroll with no URLs is invisible past the first viewport. Render numbered pagination in the initial HTML, then upgrade it with JavaScript for scroll behavior. Every page keeps a fetchable URL. See javascript-seo.

PatternCrawlableUser experience
Numbered paginationYesOne reload per page
Infinite scroll, no URLsNoSmooth, unindexable past page 1
Numbered pagination upgraded to infinite scrollYesSmooth, every page has a URL

Avoid these errors

  • rel="canonical" from page 2 and beyond back to page 1.
  • Infinite scroll with no fallback URLs.
  • Millions of crawlable filter permutations (?color=red&size=10&material=leather&sort=price).
  • Blocking a facet in robots.txt and also adding noindex; the block hides the tag.
  • Session IDs in paginated URLs, which mint fresh “pages” per session.

Backlinks