What each file actually does
Robots.txt is a plain text file of crawling rules. It lists paths a crawler should not visit: the admin area, cart, internal search, sorting parameters. The goal is to stop robots from wasting effort on service pages so they reach the useful ones sooner.
Sitemap.xml works the other way round. It is a list of addresses you want to appear in search. It helps most on new sites with few inbound links and on large catalogues where some pages sit deep in the structure.
One caveat: robots.txt does not remove a page from search results. A blocked URL can still show up if other pages link to it. To keep a page out of the index you need a noindex tag. And a sitemap is a hint, not a command: the crawler reads it, then makes its own decision.
Linking the sitemap from robots.txt
You point to the sitemap with a separate line in robots.txt: Sitemap: https://example.uz/sitemap.xml. The address must be complete, with protocol and domain. The line is independent of any User-agent block and is read by all crawlers. If you have several sitemaps, give each its own line or reference one index file that lists the rest.
This line does not replace submitting the sitemap in Google Search Console and Yandex Webmaster. Those dashboards show processing status, the number of discovered URLs and errors. The robots.txt line is a safety net for other search engines and tools.
What to block and what to include
Take an online appliance store in Tashkent. The cart, checkout, customer account, internal search and sorted listings are sensible candidates for blocking. The sitemap should contain only pages a visitor would actually want to land on:
- the home page
- categories and subcategories
- product pages that are currently on sale
- brand pages
- blog articles and reviews
- contact and delivery terms pages
Where the two files contradict each other
Sitemap generators do not know what you blocked in robots.txt, so conflicts tend to creep in quietly. These do not belong in a sitemap:
- URLs blocked with Disallow
- pages carrying a noindex tag
- addresses that redirect elsewhere
- pages returning 404 or a server error
- duplicates whose canonical tag points to another URL
- parameter, trailing-slash and non-trailing-slash variants of the same page
How to check that both files agree
Done in order, the check is quick.
- Export every URL from sitemap.xml.
- Run the list through a crawler: each address should answer with a 200 status.
- Confirm that none of them match a Disallow rule. Yandex Webmaster has a robots.txt analyzer where you can paste a list of URLs.
- Check that the pages have no noindex and that the canonical tag points to the page itself.
- Open the sitemap report in Search Console and look for the status “blocked by robots.txt”.
Changing sections and running bilingual sites
When you close a section, remove its URLs from the sitemap first, add noindex, wait until the pages drop out of the index, and only then add Disallow if you still need it. Blocking in robots.txt straight away stops the crawler from ever seeing the noindex tag, so pages can linger in results. When opening a section, reverse the order: lift the block, then add the URLs to the sitemap.
Bilingual sites need extra care. If your pages live in /uz/ and /ru/ folders, both versions should be in the sitemap and the robots.txt rules should treat them equally. It is common to see internal search blocked in the Russian version while its Uzbek twin was forgotten. The same goes for CMS plugins, which can add tags and attachments to the sitemap on their own, so review both files after every update.
If you would rather hand this check to someone else, markdev runs technical audits as part of SEO optimisation in Uzbekistan. Tell us about your site and we will talk through what makes sense for it.
FAQ
Do I need the sitemap line in robots.txt if I already added it in Search Console?
It is not required, but it helps. The line lets other search engines and tools find the sitemap without manual submission, and it does no harm.
Can I block a page in robots.txt and keep it in the sitemap?
You should not. You are asking the crawler both to find the page and to stay away from it. Search Console will flag those URLs as blocked, and trust in the sitemap drops.
How do I remove a page from search?
Take it out of the sitemap, add noindex and wait until it leaves the index. Add Disallow later, and only if needed, otherwise the crawler cannot see the tag.
How often should I review both files?
Include the comparison in your regular technical checks, and repeat it after installing plugins, changing the site structure or launching a new section.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.