A site with several hundred pages, half of which do not appear on Google, is a common problem. The frequent cause: the search engine simply cannot find these URLs. The XML sitemap comes into play here, providing crawlers with a structured list of pages to crawl.
But placing a sitemap file at the root of the site is not enough. The way it is constructed, what is included (and what is excluded) directly determines its effectiveness for indexing and visibility.
XML Sitemap: What Google Really Uses in the File
It is often said that you need to provide the priority, change frequency, and last modified date for each URL. In practice, Google ignores the priority and changefreq values. Maintaining these fields does not contribute anything to SEO on this engine.
The only parameter that really matters on Google’s side is lastmod, provided it reflects a real change in content. Changing a word in the footer or updating the copyright year does not constitute a significant change. Google checks the consistency of this date: if it is artificially updated with each generation of the sitemap, the signal loses all reliability.
For lastmod to be useful, it must correspond to an update of the main content, structured data, or internal links of the page. For example, you can check the Flow3 site to see how a well-structured sitemap organizes its URLs with consistent dates.
A commonly misunderstood point: submitting a URL in a sitemap does not guarantee its crawling or indexing. Google treats the file as a signal, not as a directive. If a submitted page remains absent from the index, the problem likely lies with the quality or uniqueness of the content, not the sitemap itself.
Filtering Sitemap URLs to Guide Crawling
The common reflex is to include all pages of the site in the sitemap. This is a mistake. A sitemap that lists noindex URLs, pagination pages, search filters, or duplicates sends a bad signal to crawlers.
The goal is to keep only the canonical, crawlable, and indexable URLs. Each URL present in the file must meet three criteria:
- It returns a HTTP 200 code (no 301 redirects or 404 or 410 errors)
- It corresponds to the canonical version of the page (the canonical tag points to itself)
- It is not blocked by the robots.txt file or a noindex tag
On an e-commerce site or a portal with thousands of pages, the sitemap is segmented into several thematic files (products, categories, blog articles). This segmentation makes it easier to identify, in Google Search Console, which group of pages is causing indexing issues.
Submitting and Monitoring the Sitemap in Search Console
Placing the file at votresite.com/sitemap.xml is a first step. But without explicit submission via Google Search Console, the engine may take time to detect it. Manual submission speeds up the initial consideration.
Once submitted, Search Console displays the number of discovered URLs and the number of URLs actually indexed. The gap between these two figures is the true performance indicator of the sitemap. A large gap signals either low-quality pages or technical issues (server response time, duplicate content, thin pages).
Regularly checking the sitemap indexing report allows for detecting regressions after a migration or redesign. During a site migration, poorly redirected URLs may remain in the old sitemap, generating a series of 404 errors in the report.
IndexNow and Sitemap: Two Complementary Mechanisms for Bing
The sitemap works in passive mode: you place a file, and the engines come to read it according to their own schedule. For sites that frequently publish or modify content, this delay can be a hindrance.
The IndexNow protocol, supported by Bing, allows you to notify the engine in real-time that a URL has changed. Instead of waiting for the bot to come back and crawl the sitemap, you send an immediate ping. This is particularly useful for news sites, marketplaces, or directories with listings updated daily.
Google has not adopted IndexNow to date. For this engine, the combination of XML sitemap plus indexing API (for eligible content types) remains the most effective combination. The strategy therefore varies according to the targeted engines, and a site aiming for both Google and Bing should combine both approaches.
- XML sitemap updated with reliable lastmod dates for Google
- IndexNow activated to signal changes in real-time to Bing
- Submission in the webmaster tools of both engines (Search Console and Bing Webmaster Tools)
Common Errors That Neutralize a Sitemap
Some technical errors render the sitemap useless without the webmaster realizing it. The most common: a robots.txt file that blocks access to the sitemap itself. The file must contain the Sitemap: directive followed by the complete URL of the file.
Another trap: exceeding the size limit. A single sitemap cannot contain more than 50,000 URLs or exceed 50 MB uncompressed. Beyond that, you need to create a sitemap index file that references multiple child sitemaps.
Finally, leaving an outdated sitemap online after a redesign creates confusion. Old URLs generate errors in Search Console, dilute crawl budget, and slow down the consideration of new pages. After each structural change to the site, regenerating and resubmitting the sitemap should be part of the deployment protocol.
The sitemap remains a technical file that is simple to produce. Its value lies less in its creation than in its maintenance: clean URLs, reliable dates, regular monitoring in Search Console. It is this rigor that transforms a passive XML file into a concrete visibility lever.
