How often should I regenerate my sitemap?

Whenever the set of URLs changes, which for most sites means at build time. Regenerating on a schedule with fresh timestamps on unchanged pages is worse than not regenerating at all, because it destroys the value of lastmod.

Should I include images and videos?

Only if media search matters to you commercially. For most sites it adds size and maintenance for no measurable return. The default answer is no.

My sitemap says submitted but not indexed. What now?

That is the normal state for a large share of any site’s URLs, and it means the engine found the page and did not consider it worth keeping. The fix is not sitemap work. It is either internal links pointing at the page, or accepting the page has no demand behind it.

What a sitemap is actually for, and what it will not fix

You have a sitemap and you want to know whether it is doing anything, or whether it needs to be bigger.

Short answer

A sitemap is a discovery aid: it tells a crawler which URLs exist and roughly when they changed. It does not make pages rank, does not guarantee indexing, and does not compensate for pages nothing links to. An accurate sitemap listing only canonical, indexable, 200-returning URLs is worth more than an exhaustive one full of redirects and duplicates.

What it is for

Discovery, and only discovery. A crawler that already reaches every page through your internal links learns almost nothing from your sitemap. A crawler that cannot reach a section because nothing links to it learns that the section exists, which is genuinely useful for new sites and deep archives.

What belongs in it

Canonical URLs that return 200 and are indexable. That is the whole rule, and applying it strictly removes most of what people put in sitemaps: redirected URLs, parameter variants, paginated duplicates, noindexed pages, and anything blocked in robots.txt. Every one of those teaches the crawler that your sitemap is unreliable.

Why accuracy beats completeness

A sitemap is a claim about your site. If a meaningful share of the URLs in it redirect or 404, the crawler learns to weight the whole file less, including the entries that were correct. A short accurate file is treated better than a long sloppy one, which is the opposite of how most people build them.

lastmod, and the way it gets ruined

lastmod is useful only if it is honest. Regenerating your sitemap nightly with today’s date on every URL tells the crawler nothing, and once it notices the dates do not correspond to real changes it stops using them. Set lastmod from actual content modification or leave it out.

Priority and changefreq

Effectively ignored by Google. Setting every page to priority 1.0 does not raise anything, and the field has been publicly described as not used. Harmless to include, pointless to agonise over, and a good sign that a sitemap tutorial you are reading is out of date.

Submitting it, and where

A Sitemap: line in robots.txt covers every crawler that looks. Add it to Search Console and Bing Webmaster Tools as well, because both then report coverage against it, and that report is the actual reason to care: it tells you which submitted URLs are indexed and which are not.

The failure it cannot fix

A URL that is in your sitemap, crawlable, and linked from nothing will often sit unindexed anyway. Being listed is not the same as being considered worth keeping. If a page matters, link to it from a page that already gets crawled.

Questions

How often should I regenerate my sitemap?: Whenever the set of URLs changes, which for most sites means at build time. Regenerating on a schedule with fresh timestamps on unchanged pages is worse than not regenerating at all, because it destroys the value of lastmod.
Should I include images and videos?: Only if media search matters to you commercially. For most sites it adds size and maintenance for no measurable return. The default answer is no.
My sitemap says submitted but not indexed. What now?: That is the normal state for a large share of any site’s URLs, and it means the engine found the page and did not consider it worth keeping. The fix is not sitemap work. It is either internal links pointing at the page, or accepting the page has no demand behind it.

Measured, not asserted

ihatepdf.cv reached 100,542 users with a sitemap listing well under two hundred URLs, one entry per real page. Of 160 pages in the final dataset, 93 of 160 earned fewer than ten clicks: being listed in the sitemap did nothing for them, because listing has never been the constraint.

Free tool for this: Internal Link Graph. No account, nothing uploaded.

Where this goes deeper

Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 10 and Chapter 17 of the book. Five chapters are free to read.