Why a sitemap goes bad on its own
A sitemap is a list of URLs you ask search engines to crawl. The generator in your CMS builds it by its own rules and never asks whether a page still matters. Over time the file collects products that were taken off sale, test pages, old URLs from before a restructure and links with tracking parameters.
Picture an online shop in Tashkent: the range changes every week, some product cards are deleted, and the sitemap keeps offering them to Google. The crawler wastes visits on dead addresses and comes less often to the pages that bring orders. Check the file after launch, after a CMS or domain move and after any structural change; in calm periods, do it at least once a quarter.
Step 1. Check the sitemap for access errors
Open the sitemap URL in your browser. It should load straight away, with no redirect and no login. If you see a normal web page or a 404 instead of XML, search engines cannot read it either.
Then open robots.txt and find the Sitemap line. The path there must match the real address exactly, including the protocol and the www or non-www version. A common cause of trouble is a sitemap that was moved while robots.txt still points to the old location.
Step 2. XML syntax
XML is strict. One small slip can make a search engine drop the whole file or part of its entries. If the browser shows a tree of tags with no error message, the basic format is fine; for extra certainty, run the file through an online XML or sitemap validator.
- an empty line or space before the XML declaration at the top of the file
- an unescaped ampersand in URLs with parameters
- unclosed tags after manual editing
- an encoding other than UTF-8, or non-Latin characters in URLs that are not encoded
- PHP error messages or debug output inside the file
Step 3. What is actually inside the file
A valid file is not necessarily a useful one. Export the URLs and load them into a crawler in list mode, such as Screaming Frog or a similar tool. Look at the status code, indexing directives and canonical tag of each address.
Only pages you want to see in search should stay in the sitemap. Anything else sends a contradictory signal: you ask for a URL to be indexed while blocking or redirecting it.
- URLs that redirect (301, 302): replace them with the final address; remove 404 and 410 pages
- pages with noindex or blocked in robots.txt
- URLs whose canonical points to a different page
- duplicates: http and https, www and non-www, with and without a trailing slash
- URLs with UTM tags from Instagram or Telegram ads, and sorting parameters
- utility pages: cart, account area, the hand-off to Click or Payme payment
Step 4. The reverse check: what is missing
Compare the URLs found by crawling your site with the URLs in the sitemap. If a new section or category exists on the site but not in the file, the generator settings are the likely cause, for example it may only include certain content types.
If the site has Russian and Uzbek versions, both should appear in the sitemap with consistent URLs. The lastmod date should reflect a real content change, not the moment the file was generated, otherwise search engines learn to ignore it.
Step 5. Search Console and Yandex Webmaster
In the Sitemaps section of Google Search Console you can see when the file was last read, how many URLs were found and whether there are errors. A useful trick: limit the indexing report to URLs from your sitemap and see which of them are not indexed, and why.
Yandex Webmaster has a list of sitemap files and an analysis tool that reports format errors. Data does not update instantly, so after fixing the file, resubmit it and come back to the report later. Compare the number of URLs in the panel with the number in the file; a gap is a reason to dig.
On larger sites it helps to split the sitemap by page type, such as products, categories and articles, and tie the parts together with an index file. The reports then show at once which section has indexing problems.
Who should check, and how often
The owner can open the file and read the reports. Exporting URLs and checking server responses is easier to hand to an SEO specialist or developer. What matters is that the check belongs to someone's routine and doesn't start only after traffic drops. If errors keep coming back after you fix them, repair the generator settings, not the file.
If you would rather hand the sitemap audit and other technical fixes to a team, markdev provides SEO optimisation in Uzbekistan. Describe your task and we will suggest where to start.
FAQ
How often should I check my sitemap?
Right after launch, a domain or CMS move and any big structural change. In normal times, glance at the Search Console and Webmaster reports monthly and run a full check at least once a quarter.
Can I check a sitemap without special tools?
The basics are manual: open the file in a browser, confirm it loads without a redirect and read the status in the search engine panels. To verify response codes for every URL you will need a crawler.
Which pages should be removed from a sitemap?
Redirecting and broken URLs, pages with noindex, pages blocked in robots.txt, non-canonical duplicates, URLs with UTM tags or parameters, and utility pages such as the cart and account area.
Why does Search Console show fewer URLs than my file contains?
The file may not have been re-read yet, some entries may have been dropped because of format errors, or the file may contain duplicates. Check the last read date and the error list first.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.