XML Sitemaps: The Honest Guide to Getting Every Page Found (Plus the Free Generator I Built)

What is an XML sitemap, in plain language?

An XML sitemap is a machine-readable file that lists the important URLs on your website so search engine crawlers can find them. It lives at your site root, usually at yoursite.com/sitemap.xml. Each entry contains the page URL and, optionally, the date it last changed. Googlebot and other crawlers use it as a backup discovery method: a second route to your pages beyond following links.

Think of your website as a building and Googlebot as a visitor. Internal links are the corridors. A visitor who follows the corridors eventually sees most rooms. An XML sitemap is the directory by the front door. It lists every room that matters, including the ones tucked down corridors the visitor might never wander into. Nobody reads it for fun; it exists for robots. It is written in a format called XML, which is just structured text with labels that machines understand. One file. No design. No visitors. Just a list that says: here are my pages, and here is when each one changed.

When does a sitemap actually matter, and when is it pointless?

A sitemap matters most for new sites, large sites, and sites with weak internal linking. If your pages sit several clicks deep, have no incoming links, or number in the thousands, a sitemap gives crawlers a second route to find them. If your site has fifty well-linked pages and real traffic, Google discovers everything through normal crawling, and the sitemap is mostly insurance.

New sites are the clearest case. Google has no history with you. Your internal links are fresh and your backlinks are few. A sitemap is the one signal you control on day one that says "these pages exist." Large sites are the second case. Once you pass a few thousand pages, crawlers will not reach everything through links alone. Orphan pages are the third case: the service page nobody linked to from the navigation, or the blog post from 2022 that fell off the homepage widgets. The sitemap is their second chance.

Now the part nobody selling a generator will say. If you run a small business site with thirty pages, clean navigation, and a footer that links to everything, Google will find all of it without any sitemap. I have seen sites like this rank perfectly well with no sitemap at all. You should still have one. It takes minutes and costs nothing. But it is insurance, not fuel.

Does submitting a sitemap guarantee Google will index my pages?

No. A sitemap is a hint, not a command. Google has said this in plain language for years. Submitting a sitemap does not guarantee that the file will be downloaded, that the URLs will be crawled, or that the pages will be indexed. Indexing is a separate decision based on page quality, uniqueness, and how much crawl attention your site has earned.

This is the misunderstanding I see most. Someone submits a sitemap, waits a week, and asks why half the pages still show as "Discovered - currently not indexed." The sitemap did its job. Google looked at the list and decided which pages deserved a visit, and some were not worth it yet. There is a sharper edge to this, too. Google's Search Relations team has explained on its Search Off the Record podcast that Google may not bother fetching sitemaps on sites where crawl demand is weak, such as sites it considers low quality. A technically correct file does not guarantee anyone reads it.

So keep expectations honest. A sitemap raises the ceiling on discovery. It does not raise the floor on quality. If Google does not want to crawl your site, the fix is a better site, not a better XML file.

What belongs in an XML sitemap, and what should never go in one?

An XML sitemap should contain only the URLs you actively want indexed: canonical pages that return a clean 200 status. It should never contain noindex pages, redirected URLs, 404 errors, pages blocked by robots.txt, non-canonical duplicates, or thin pages you would not show a visitor. Every wrong URL in the file is a conflicting signal that wastes your crawl budget.

The conflicting-signal problem is the real killer. Your sitemap says "please crawl this page." The page carries a noindex tag that says "please do not index this page." Google reads both and flags the inconsistency. It is the same story when you list a URL whose canonical tag points somewhere else, or a URL that redirects. You are sending directions and roadblocks in the same envelope.

The biggest mistake I see is dumping everything in. A WordPress SEO plugin generates its sitemap from your site's routes, which means tag pages, attachment pages, author archives, and search result pages can all end up listed. "Generated from routes" does not mean "worth indexing." Before any sitemap goes live, the URL list deserves a human review. If you would not proudly submit a page to Google, it does not belong in the file.

What is a sitemap index file, and when do you need one?

A sitemap index file is a parent file that points to multiple child sitemap files. You need one when a single sitemap hits the limits: 50,000 URLs or 50 MB uncompressed per file. Instead of one giant file, you split URLs into smaller sitemaps and list them all in the index. You submit the index to Search Console, and Google works through each child file.

The limits sound generous until you run an ecommerce store or a multi-location directory. No free online XML sitemap generator will crawl a 50,000-page store for you. The answer is not a bigger crawl button. It is a sitemap index. Here is how it works in practice. Split by content type: products in one child file, blog posts in another, location pages in a third. Each child stays well under the limits and is easier to debug. When the products file throws an error in Search Console, you know exactly where to look.

The index itself is a tiny file. It lists the locations of the child sitemaps, something like yoursite.com/sitemap-products.xml and yoursite.com/sitemap-posts.xml. You can also reference the index in your robots.txt with one line. One more practical note: keep each child comfortably below the ceiling. A file with 49,900 URLs works. A file with 50,001 breaks the limit. Build with margin.

How is a sitemap different from robots.txt?

A sitemap and robots.txt do opposite jobs. The sitemap is an invitation: here are my pages, please come look. Robots.txt is a gatekeeper: these paths are off limits, do not enter. They live side by side in your site root, and they must never contradict each other.

The classic contradiction goes like this. Someone lists a URL in the sitemap and blocks the same path in robots.txt. The sitemap invites Googlebot in, and the gatekeeper turns it away at the door. Search Console will flag this as a conflict, and you are sending Google mixed signals. Another common tangle is blocking a page in robots.txt while expecting Google to obey its noindex tag. If the crawler cannot visit the page, it never sees the noindex instruction, and the page can sit in the index as a bare URL with no description. Robots.txt controls crawling. Meta tags control indexing. They are different systems, and mixing them up is one of the most expensive beginner mistakes in technical SEO.

How do you actually create an XML sitemap, step by step?

Creating an XML sitemap takes six steps. Audit which of your pages are actually indexable, pick a generator that fits your platform, configure exclusions for junk URLs, run the crawl, review the URL list before exporting, and upload the file to your site root. The step everyone skips is reviewing the URL list. That review is the difference between a useful sitemap and a bloated one.

Step one: walk your own site. Decide which pages you want in search results: service pages, location pages, blog posts, key category pages. Write the list down. Step two: pick your method. On WordPress, Yoast or Rank Math generates one automatically. On Shopify, Wix, or Squarespace, one already exists. For anything else, a free online XML sitemap generator does the job with no install.

Step three is where the pros separate from the tourists. Configure exclusions in your XML sitemap generator before you crawl. Block your /tag/ and /category/ pages if they add nothing. Block /wp-admin/, search result pages, and any pattern that produces thin duplicates. Step four: run the crawl. Step five: read the URL list it produces. This is the review step. Remove anything you would not want a stranger to land on from Google. Step six: download the file, upload it to your site root as sitemap.xml, and check that it loads in your browser.

How do you submit your sitemap to Google Search Console?

Open Google Search Console, select your property, click Sitemaps in the left menu, enter the file name (for example, sitemap.xml), and click Submit. Google validates the file and starts working through it, often within a day or two. Full indexing of new pages takes longer, typically a few weeks, depending on your site's authority.

After submitting, watch the Sitemaps report. It tells you how many URLs Google discovered in the file and flags errors such as "Couldn't fetch," invalid format, or URLs it could not reach. If you see "Couldn't fetch" on a valid file, the usual causes are server load or low crawl demand, not a broken file. One habit to drop: do not resubmit the sitemap every day. Google rechecks it on its own schedule, and resubmitting does not move you up any queue. I know because I once resubmitted daily for a week on a client site and changed absolutely nothing except my own stress level. Submit once, read the report, fix real problems, and let the schedule do its work.

For Bing, the path is the sitemap submission in Bing Webmaster Tools. The same file works for both.

What do the lastmod, priority, and changefreq tags actually do?

For Google, almost nothing. Google officially ignores the priority and changefreq tags. It uses only lastmod, and only when the date is consistently and verifiably accurate. Many generators still fill in priority scores by default, but for Google those numbers are decoration.

Priority was designed to tell crawlers which pages matter most, on a scale from 0.0 to 1.0. Changefreq was supposed to say how often a page changes. Google has said plainly, more than once and in its own documentation, that it ignores both. The fields still exist in the sitemap protocol, and non-Google crawlers may glance at them, which is why generators keep filling them in. But no priority value changes how Google treats a page.

Lastmod is the one tag with real weight. Google uses it to decide which pages deserve a fresh crawl. The catch is accuracy. Google only trusts lastmod when it is consistently and verifiably correct. Many one-off generators stamp today's date on every URL, including pages that have not changed in a year. That is noise, and Google learns to ignore it. An honest lastmod reflects the last meaningful content change. A missing lastmod is more accurate than a fake one. This is the kind of detail a practitioner learns by watching crawl stats, not by reading an XML sitemap generator's feature list.

How often should you regenerate or update your sitemap?

Regenerate your sitemap whenever you add pages, delete pages, or significantly update content. For active sites, that means at least monthly. For blogs that publish weekly, it means weekly. The real answer depends on how your sitemap is built, because static files rot and dynamic ones do not.

This is the stale-file problem nobody's sales page mentions. You run a free generator once, upload a beautiful sitemap.xml, and forget about it. Six months later it lists twelve pages you deleted and misses forty you published. Google keeps checking the old list, and your new content waits in line behind dead URLs.

There are two honest fixes. The first is to use a generator that updates automatically. WordPress plugins like Yoast and Rank Math regenerate the sitemap the moment you publish, edit, or delete. That is dynamic generation, and it is the best option for most sites. The second is to build a regeneration habit. Pick a schedule, put it on the calendar, and actually do it. Set a reminder for the first Monday of the month. It takes ten minutes. The worst option is the beautiful one-time file that slowly becomes fiction.

What does a clean, minimal sitemap.xml look like?

A clean sitemap contains only loc and an honest lastmod for each URL, uses the correct XML namespace, and lists nothing you would not want indexed. Here is the pattern. Every extra tag beyond this is optional decoration.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/ac-repair-las-vegas/</loc>
    <lastmod>2026-09-28</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/why-your-ac-freezes-up/</loc>
    <lastmod>2026-10-02</lastmod>
  </url>
</urlset>

That is it. Two entries, each with a URL and the date its content last meaningfully changed. Compare this with the bloated version many generators produce: every URL carrying priority 0.5, changefreq weekly, and a lastmod of today. The file is several times larger and carries far less truth. Smaller and honest beats bigger and decorative. If you only know one page's real change date, give that page a lastmod and leave the others without one rather than faking it. That is the standard.

The best XML sitemap generator tools in 2026, compared honestly

For most sites, the best generator is the one built into your platform. WordPress users should use Yoast or Rank Math. Shopify, Wix, and Squarespace already make one for you. Everyone else should choose between a free online generator for a quick one-off file and Screaming Frog when they need to see the crawl before exporting.

My methodology is simple. I ran the free XML sitemap generator options, read their output, and asked one question: does this file help, or does it just exist? The category has consolidated hard. The platforms generate sitemaps themselves now, and Screaming Frog covers the crawl-first cases. Recommending a standalone paid sitemap tool in 2026 is padding, not advice. For a one-off file on a non-WordPress site, a free online XML sitemap generator is the right tool, used once and then on a schedule.

Why are my sitemap pages still not indexed?

Because the sitemap only handles discovery. If pages are discovered but not indexed, the problem is the page, not the file. Run five checks: confirm the sitemap has no errors in Search Console, inspect a sample URL to see Google's actual verdict, look for canonical conflicts, check for noindex pages leaking into the file, and be honest about content quality.

Start with the Sitemaps report. Zero errors there means the file is fine and the bottleneck is elsewhere. Use URL Inspection on a few stubborn pages. Google tells you directly: crawled but not indexed, duplicate without user-selected canonical, or blocked. Each verdict calls for a different fix.

The two sneakiest causes are canonical conflicts and thin content. A canonical conflict means your sitemap lists URL A while your canonical tag points to URL B. Google uses the sitemap as one input into canonical selection, then picks B, and A sits unindexed forever. Fix the mismatch, not the sitemap. Thin content is the other one. A sitemap full of tag archives and auto-generated pages dilutes the signal, so prune it. If a page has been live for a month, is internally linked, passes inspection, and still is not indexed, waiting longer will not help. Improving the page will.

Why I built my own XML sitemap generator instead of relying on plugins

I built it because of a 404. When my own site's plugin-generated sitemap broke silently, I wanted a tool that crawls any site with no install, lets me exclude junk URL patterns before generating, can include image metadata, and produces a file I can review before downloading. So I built the free XML sitemap generator on LocalSEOFlow.

The design decisions came straight from my mistakes. The 500-URL crawl covers most local business sites, which run twenty to eighty pages. The exclude-patterns setting exists because of the plugin sitemap I once saw packed with tag pages and attachment pages, and it comes prefilled with /tag/, /category/, and /author/. The pre-download URL review exists because nobody checks what their generator actually listed. The optional image metadata toggle is there because Google can index your photos alongside your pages, and many free generators skip them.

Links to submit the result to Search Console and Bing Webmaster Tools close the loop. The tool also adds smart priority scores, which are useful for organizing the file and for crawlers that read them, but remember that Google ignores priority. The fields that matter to Google are loc and an honest lastmod. Everything else is thoroughness, not magic.

If the pages you need discovered are backlinks on other people's sites rather than pages on your own site, that is a different tool for a different job. The backlink indexer handles discovery for individual URLs through crawl pages and submission channels. A sitemap is your own site's map. The indexer is the knock on Google's door for pages you do not control.

The file is simple. The discipline is the hard part.

That hand-built 21-page sitemap went live with my site and worked exactly as intended. Every URL I wanted indexed was in it, and nothing I did not want was. No plugin could silently break it, because there was no plugin involved. It took twenty minutes. A free XML sitemap generator would have done the crawling in less time. Nothing would have done the thinking for me.

That is the whole lesson, really. An XML sitemap is a list. The craft is in what you put on the list, what you keep off it, and whether you keep it honest as your site changes. Google ignores your priority scores. It notices your fake timestamps. It may skip your file entirely if your site gives it no reason to care. None of that is the file's fault.

So here is what I would do this week. Open your sitemap in a browser and actually read it. Count the URLs you would not defend, and remove them. Check the lastmod dates against reality. Set one calendar reminder to regenerate it. That single hour will do more for your crawl health than any tool purchase.

When did you last look at your own sitemap.xml? Not submit it, but actually look at it. What might you find in there that would surprise you?

Frequently asked questions