LLM Best Practices

IndexNow

4 min read

View markdown source

Overview

IndexNow is an open protocol that lets a site push URL changes to participating search engines instead of waiting for them to crawl. A POST to one endpoint reaches Bing, Yandex, Seznam, Naver, and Yep at once. Google does not participate. Ship IndexNow on any site that publishes or updates content faster than once a week.

IndexNow is push, not pull

Traditional indexing is poll-based: the engine decides when to recrawl. IndexNow inverts that.

  • Site publishes or updates a URL.
  • Site POSTs the URL list to an IndexNow endpoint.
  • Participating engines receive the change within seconds and queue a recrawl within minutes.
  • Submitted URLs still go through normal quality and ranking pipelines; submission is not a guarantee of indexing.

IndexNow does not replace a sitemap. Sitemaps remain the authoritative full inventory; IndexNow is the delta channel on top. Both belong on the same site, alongside the canonical and URL hygiene rules in technical. For sites above 10k URLs, IndexNow also cuts the latency cost of crawl-budget discovery.

Bing and Yandex participate; Google does not

The current participant list, in rough order of search-market relevance:

  • Bing (and any engine that licenses Bing’s index, including DuckDuckGo and Ecosia).
  • Yandex.
  • Seznam (Czech Republic).
  • Naver (South Korea).
  • Yep (Brave’s index).

Google has publicly declined to join. For Google, rely on Search Console’s URL Inspection and on the sitemap’s <lastmod> plus normal crawl. Do not skip IndexNow because Google is absent; Bing-derived traffic alone justifies the integration, and AI search products (Bing Chat, Perplexity, Copilot) lean on Bing’s index.

Host the key file at the site root

Authentication is by a shared secret served as a static file.

  • Generate a random hex string at least 8 and at most 128 characters. UUID without hyphens works.
  • Host it at https://<your-domain>/<key>.txt with the file contents equal to the key.
  • Serve with Content-Type: text/plain and a 200 status.

The engine verifies ownership by fetching the key file before honoring submissions. Rotate the key only when it leaks; rotation invalidates pending submissions.

POST URL lists to the endpoint

Two shapes. The single-URL form is a GET; the batch form is a POST with JSON.

Batch form, up to 10,000 URLs per request:

curl -X POST "https://api.indexnow.org/IndexNow" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "example.com",
    "key": "<your-key>",
    "keyLocation": "https://example.com/<your-key>.txt",
    "urlList": [
      "https://example.com/seo/technical",
      "https://example.com/seo/content"
    ]
  }'

Response codes worth wiring into alerting:

  • 200 or 202: accepted.
  • 400: malformed request. Fix the payload.
  • 403: key file missing, wrong contents, or wrong URL.
  • 422: URLs do not match the declared host. Filter the list.
  • 429: rate-limited. Back off.

Submit only on real changes

Submitting unchanged URLs trains engines to ignore the channel and risks throttling.

  • Submit on publish, on material update, and on delete (so the engine drops the URL).
  • Do not submit on every build. Compare against the last-deployed manifest and submit only the diff.
  • Pair the trigger with the last_updated rule in content: if the page is not re-dated, it should not be submitted.

Rate limits are not publicly fixed; the practical ceiling is in the low thousands of unique URLs per day per site.

CMS plugins cover the common case; custom integration covers the rest

For WordPress, Wix, Duda, and Cloudflare-fronted sites, an official plugin or one-click integration ships submissions automatically. For custom stacks (Next.js, Astro, Quartz), wire submission into the deploy step.

  • Generate the key once, commit the key file to the repo (or to static/).
  • After the deploy succeeds, compute the URL diff against the previous build manifest.
  • POST the diff to api.indexnow.org/IndexNow from CI.
  • Log responses; alert on non-2xx for two builds in a row.

Graph View

Backlinks