LLM Best Practices

How to submit a new site to search engines

2 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

A new site with no inbound links can sit undiscovered, so register it explicitly with three services: Google Search Console, Bing Webmaster Tools, and IndexNow. This is phase 5 of launch-a-new-site. Run it only after the site is live on its final domain with a valid certificate and a sitemap.xml that lists final-domain URLs; a sitemap of *.github.io URLs or one that 404s wastes the first crawl.

Steps

1. Verify Google Search Console and submit the sitemap

Create a Domain property and verify it with a DNS TXT record; it covers every protocol and subdomain. Then submit sitemap.xml under Sitemaps and URL-inspect the homepage and click Request indexing. The old google.com/ping?sitemap= endpoint stopped working in late 2023; the Sitemaps report and the Sitemap: line in robots.txt are the supported paths. Report details: master-google-search-console.

2. Import the property into Bing Webmaster Tools

Choose “Import from Google Search Console” to carry over the property and sitemap in one step, or verify manually and submit the sitemap. Bingbot feeds Bing and Copilot, so the Bing index is where Copilot grounding starts.

3. Deploy an IndexNow key

Host a text file at https://example.com/<key>.txt whose content is the key (8 to 128 characters: letters, digits, dashes). Submit changed URLs:

curl "https://api.indexnow.org/indexnow?url=https://example.com/new-page&key=<key>"

Or POST up to 10,000 URLs as JSON with host, key, and urlList. A 200 or 202 means accepted; 403 is a bad key and 422 a URL that does not match the host. Participating engines share submissions with each other, but Google does not use IndexNow, so rely on the sitemap and Search Console for Google. Automate the ping in a post-deploy step: indexnow.

Verify it worked

Within days, Search Console’s Sitemaps report shows Success and pages move to Indexed in the Page indexing report; Bing’s sitemap status and URL Inspection show the same.

curl -s https://example.com/<key>.txt     # prints the key

Common errors

  • Pages stay in “Discovered - currently not indexed”. That is a quality or crawl-budget signal, not a submission failure; see the indexing section of master-google-search-console.
  • IndexNow returns 403. The key file is missing at the site root or its content differs from the submitted key.
  • Sitemap lists the old host. Fix baseUrl and rebuild before submitting.

Backlinks