IndexNow & XML Sitemaps: Faster Discovery of New Pages

IndexNow & XML Sitemaps: Faster Discovery of New Pages

You publish a page, and it appears in search results two days later, or two weeks later, or not at all for a while. Sometimes the delay is about quality. Often it is about discovery: the search engine has not yet found the URL, or has found it and not yet come back to check it.

Two tools help shorten that gap. An XML sitemap lists your URLs and, when set up correctly, tells search engines which pages are new or recently changed. IndexNow is a protocol that lets your site notify participating search engines directly when a URL is created, updated, or deleted.

Neither tool is magic. A sitemap full of inaccurate dates teaches search engines to ignore it. IndexNow does not guarantee indexing. But used properly, both reduce the time between publishing a page and the search system knowing about it. This guide covers how to set them up correctly and how to check that they work.

What an XML sitemap does and does not do

An XML sitemap is a file that lists the URLs you want search engines to know about. It sits at a predictable address, such as /sitemap.xml, and it is referenced in robots.txt or submitted through Search Console and Bing Webmaster Tools.

A sitemap helps in three ways:

• Discovery. It gives search engines a list of URLs they might not find through links alone, such as new pages with few internal links.
• Change signals. The lastmod value tells search engines when a page was last modified, which can prompt a recrawl.
• Structure. Large sites can split sitemaps by type, such as products, articles, and categories, which helps monitoring.

A sitemap does not guarantee that a page will be crawled or indexed. It is a hint. Google has said that it uses the lastmod date when it is consistently accurate, and that it ignores the priority and changefreq values. Check current Google documentation before relying on any specific sitemap behaviour.

Step 1: Get the basics of a valid sitemap right

A valid XML sitemap follows the sitemap protocol. Each URL is wrapped in a <url> element, with a <loc> value containing the full, canonical URL.

<?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">  <url>    <loc>https://example.com/blog/indexnow-xml-sitemaps-faster-discovery</loc>    <lastmod>2026-10-17</lastmod>  </url>  <url>    <loc>https://example.com/services/local-seo</loc>    <lastmod>2026-09-02</lastmod>  </url> </urlset>

Several rules matter:

• Only canonical URLs. Do not list URLs that redirect, return errors, or are marked noindex. Each listed URL should be one you want indexed.
• Full, absolute URLs using the protocol and domain your site actually uses.
• Consistent protocol and host. Mixing http and https, or www and non-www, sends mixed signals.
• Size limits. A single sitemap file is limited to 50,000 URLs and 50 MB uncompressed. Split larger sites into multiple files referenced by a sitemap index.
• Valid encoding. Escape special characters such as ampersands in URLs.

Check the file with a validator, and open it in a browser to confirm it renders.

Step 2: Make lastmod accurate, or leave it out

The lastmod element is the most valuable and the most abused field in a sitemap. Search engines use it to decide when to recrawl. If you set it to today's date on every URL every time the sitemap regenerates, the signal becomes meaningless, and search engines learn to distrust it.

Set lastmod to the date the page's main content last changed, not the date the sitemap was built. Update it only when something meaningful changes: the text, the product data, or the substance of the page. Do not update it for changes to sidebars, footers, or trivial markup.

Where your platform cannot provide accurate dates, it is better to omit lastmod than to fill it with a value that does not reflect real changes.

The content freshness post covers how to signal recency on the page itself. The two work together: an accurate on-page update date and an accurate sitemap date tell the same true story.

Step 3: Split and organise sitemaps for monitoring

On a small site, one sitemap is enough. On a larger one, separate sitemaps by content type makes problems easier to see. A sitemap index file can reference them all.

<?xml version="1.0" encoding="UTF-8"?> <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">  <sitemap>    <loc>https://example.com/sitemap-blog.xml</loc>    <lastmod>2026-10-17</lastmod>  </sitemap>  <sitemap>    <loc>https://example.com/sitemap-services.xml</loc>    <lastmod>2026-09-02</lastmod>  </sitemap> </sitemapindex>

Separate sitemaps let you see in Search Console how many URLs in each group are submitted and how many are indexed. A large gap in one group points to a specific problem, such as thin content or a template issue, rather than a vague site-wide one.

Step 4: Submit and reference the sitemap

Reference the sitemap in robots.txt with a single line, and submit it in Google Search Console and Bing Webmaster Tools.

Sitemap: https://example.com/sitemap.xml

Check the submission status after a few days. Search Console shows how many URLs were discovered, how many were indexed, and any errors. Errors such as redirects, blocked URLs, or non-canonical URLs in the sitemap should be fixed at the source.

Step 5: Understand what IndexNow is

IndexNow is an open protocol that lets a website tell participating search engines when a URL has been created, updated, or deleted. The site sends a request to an IndexNow endpoint, and participating engines are notified of the change.

As of the time of writing, Bing, Yandex, Naver, and Seznam participate in IndexNow. Google has tested the protocol but has not adopted it as a production signal, so do not assume that submitting to IndexNow will notify Google. Check the current list of participating engines in the IndexNow documentation before relying on it.

The practical value of IndexNow is faster discovery on the engines that participate, and a reliable record of the changes you have announced. It is a complement to sitemaps, not a replacement.

Step 6: Set up IndexNow

Setting up IndexNow involves creating a key, hosting it on your site, and sending notifications when URLs change.

The steps in general terms:

1. Generate a key. This is a random string, typically 8 to 128 characters, containing only letters, numbers, and hyphens.
2. Host the key file. Place a text file at the root of your domain named after the key, containing the key itself. Participating engines use this file to verify that you control the site.
3. Send notifications. When a URL is created, updated, or deleted, send a request to the IndexNow endpoint with the URL and the key. Batch multiple URLs in one request where possible.
4. Log every submission. Keep a record of what was sent and when, so you can match it against crawl and indexing data later.

Many content management systems and SEO plugins include IndexNow support, which handles the key and the notifications automatically. If you use one, confirm that it submits only canonical, indexable URLs, and that it does not submit URLs that are noindexed or redirected.

Check the current IndexNow documentation for the exact endpoint, request format, and rate limits, as these can change.

Step 7: Decide what to submit

Submit URLs that you want indexed and that have changed meaningfully. Do not submit every URL on the site on every deployment. Bulk submission of unchanged content wastes the signal, and some engines may limit or ignore frequent resubmissions.

Good triggers for submission:

• A new page is published and ready for indexing.
• An existing page changes in a substantive way.
• A page is removed, so the engine can drop it from the index.
• A page moves to a new URL, so the engine can process the change.

Avoid submitting URLs that are noindexed, redirected, blocked by robots rules, or returning errors. Submitting them sends contradictory signals.

Step 8: Verify that discovery is actually faster

Before concluding that the setup works, measure it.

A practical test:

1. Publish a set of new pages and record the publication time.
2. Submit half of them through IndexNow, and leave the other half to sitemap discovery only.
3. Check, every day, whether each page appears in a site search or in Search Console's URL inspection tool, and record when it first appears.
4. Compare the two groups.

Use a sample large enough to be meaningful, and remember that results vary with the site's authority, the topic, and the engine. A single test is not conclusive, but a consistent difference across several batches is useful evidence.

Track ongoing signals in Search Console: the index coverage report, the sitemap report, and crawl stats. A rising share of submitted URLs that are indexed indicates the setup is working.

Step 9: Avoid the common failure modes

Several mistakes reduce the value of both tools.

• Stale or false lastmod values. Search engines learn to ignore them.
• Listing non-canonical or redirected URLs. The sitemap becomes a list of errors.
• Submitting noindexed pages. The signals contradict each other.
• Failing to update the sitemap when pages are removed. Deleted pages keep appearing in the sitemap and generate errors.
• Relying on IndexNow to index a poor page. Notification speeds up discovery; it does not make weak content indexable.
• Forgetting the key file. Without it, participating engines cannot verify the submissions.

What these tools cannot do

Neither sitemaps nor IndexNow will rescue a page that is low quality, duplicated, or blocked. They speed up the search engine's awareness of a URL. The decision about whether to index and rank that URL depends on the page itself.

They also cannot fix a site where pages are hard to reach. A new page with no internal links and no sitemap entry is harder to find, and the internal linking strategy post covers the link side of discovery.

Common discovery mistakes to avoid

Setting lastmod to the build date. Every page appears freshly changed, and the signal is lost.

Including redirects and errors in the sitemap. Clean the sitemap to canonical, indexable URLs only.

Expecting IndexNow to reach Google. Check the current list of participating engines and plan accordingly.

Submitting everything on every deploy. Submit what has changed meaningfully.

Ignoring the Search Console sitemap report. The errors it shows are the fastest route to fixing the setup.