XML Sitemap

An XML sitemap is a file a website publishes that lists the URLs it wants crawlers to know about, each with an optional date of last significant change. It is a discovery aid: it tells a crawler what exists and what has moved, and it grants no permission and makes no promise about what will be indexed.

What goes in it

Fully qualified absolute URLs, and only the canonical version of each page, since a sitemap that lists both the desktop and the mobile address, or the page with and without a tracking parameter, is telling the crawler to choose. Google’s sitemap documentation caps a single file at 50 MB uncompressed or 50,000 URLs and allows a sitemap index file to point at several. Of the optional fields, Google states that it uses the last-modified date when it proves consistently accurate and ignores priority and change frequency altogether. A last-modified date that updates on every deploy, whether or not the page changed, is worse than none.

How a crawler finds it

Two routes. The file can be submitted directly through a search engine’s console, or its address can be published in robots.txt as a Sitemap record. RFC 9309, the Robots Exclusion Protocol, names Sitemaps as its example of the other records a crawler may read from that file beyond the allow and disallow rules. Google also notes that, unless it is submitted through Search Console, a sitemap covers only URLs beneath the directory it sits in, which is why the file lives at the site root.

What it does not do

A sitemap does not override robots.txt: a URL listed in the sitemap and disallowed for a crawler’s product token stays disallowed. It does not guarantee crawling or indexing; Google’s overview says so directly. And it does not describe the business or its pages in any way an assistant can quote. It is a list of addresses. What sits at each address, and whether the crawler receives it at all, is a separate question, which is why this site’s crawler access check reads robots.txt rather than the sitemap.

A common question

Does a small site need an XML sitemap?

Google’s documentation says a site of roughly 500 pages or fewer, with every page reachable through internal links and little media or news content, will usually be found without one. It also says that in most cases a site still benefits from having one. For a small business site the cost is minutes, and the file is a plain statement of which pages you consider yours, so there is little reason to leave it out.

Sources

Format, limits and field handling: Google’s guide to building a sitemap and its sitemaps overview. The Sitemap record in robots.txt: RFC 9309, section 2.2.4. Both read September 2026.