LLM Best Practices

IndexNow

3 min read

View markdown source

Overview

IndexNow is an open protocol that lets a site push URL changes to participating search engines instead of waiting for them to crawl. A POST to one endpoint reaches Bing, Yandex, Seznam, Naver, and Yep at once. Google does not participate. Ship IndexNow on any site that publishes or updates content faster than once a week.

IndexNow is push, not pull

Traditional indexing is poll-based: the engine decides when to recrawl. IndexNow inverts that.

  • Site publishes or updates a URL.
  • Site POSTs the URL list to an IndexNow endpoint.
  • Participating engines receive the change within seconds and queue a recrawl within minutes.
  • Submitted URLs still go through normal quality and ranking pipelines; submission is not a guarantee of indexing.

IndexNow does not replace a sitemap. Sitemaps remain the authoritative full inventory; IndexNow is the delta channel on top. Both belong on the same site, alongside the canonical and URL hygiene rules in technical.

Bing and Yandex participate; Google does not

The current participant list, in rough order of search-market relevance:

  • Bing (and any engine that licenses Bing’s index, including DuckDuckGo and Ecosia).
  • Yandex.
  • Seznam (Czech Republic).
  • Naver (South Korea).
  • Yep (Brave’s index).

Google has publicly declined to join. For Google, rely on Search Console’s URL Inspection and on the sitemap’s <lastmod> plus normal crawl. Do not skip IndexNow because Google is absent; Bing-derived traffic alone justifies the integration, and AI search products (Bing Chat, Perplexity, Copilot) lean on Bing’s index.

Host the key file at the site root

Authentication is by a shared secret served as a static file.

  • Generate a random hex string at least 8 and at most 128 characters. UUID without hyphens works.
  • Host it at https://<your-domain>/<key>.txt with the file contents equal to the key.
  • Serve with Content-Type: text/plain and a 200 status.

The engine verifies ownership by fetching the key file before honoring submissions. Rotate the key only when it leaks; rotation invalidates pending submissions.

POST URL lists to the endpoint

Two shapes. The single-URL form is a GET; the batch form is a POST with JSON.

Batch form, up to 10,000 URLs per request:

curl -X POST "https://api.indexnow.org/IndexNow" \
  -H "Content-Type: application/json" \
  -d '{
    "host": "example.com",
    "key": "<your-key>",
    "keyLocation": "https://example.com/<your-key>.txt",
    "urlList": [
      "https://example.com/seo/technical",
      "https://example.com/seo/content"
    ]
  }'

Response codes worth wiring into alerting:

  • 200 or 202: accepted.
  • 400: malformed request. Fix the payload.
  • 403: key file missing, wrong contents, or wrong URL.
  • 422: URLs do not match the declared host. Filter the list.
  • 429: rate-limited. Back off.

Submit only on real changes

Submitting unchanged URLs trains engines to ignore the channel and risks throttling.

  • Submit on publish, on material update, and on delete (so the engine drops the URL).
  • Do not submit on every build. Compare against the last-deployed manifest and submit only the diff.
  • Pair the trigger with the last_updated rule in content: if the page is not re-dated, it should not be submitted.

Rate limits are not publicly fixed; the practical ceiling is in the low thousands of unique URLs per day per site.

CMS plugins cover the common case; custom integration covers the rest

For WordPress, Wix, Duda, and Cloudflare-fronted sites, an official plugin or one-click integration ships submissions automatically. For custom stacks (Next.js, Astro, Quartz), wire submission into the deploy step.

  • Generate the key once, commit the key file to the repo (or to static/).
  • After the deploy succeeds, compute the URL diff against the previous build manifest.
  • POST the diff to api.indexnow.org/IndexNow from CI.
  • Log responses; alert on non-2xx for two builds in a row.

Graph View

Backlinks