What a robots.txt file does
A robots.txt file sits in the root of your domain (for example, your-domain.uz/robots.txt) and tells search engine crawlers which sections they may crawl and which to skip. It is not a security tool. The file is public, and well-behaved crawlers simply follow the rules it sets.
For a business the point is focus. A crawler gives your site a limited amount of attention, and you want it spent on your catalog, product pages and articles rather than on the cart, internal search and technical URLs. This matters most for online stores, where filters alone can generate hundreds of near-identical addresses.
The directives you actually need
The syntax is simple, and most sites get by with four commands:
- User-agent: * applies the rules to all crawlers; you can name a specific one such as Googlebot
- Disallow: /admin/ forbids crawling of a section
- Allow: makes an exception inside a blocked section
- Sitemap: https://your-domain.uz/sitemap.xml points crawlers to your sitemap so they find important pages faster
What is sensible to block
Block what will never bring search traffic and only clutters crawling. Picture a clothing store in Tashkent that takes payments through Click and Payme: customers need the cart and checkout, but a search engine does not.
- the admin panel and other service areas
- the cart and checkout pages
- customer account pages
- internal site search results
- URLs with sorting or tracking parameters, such as those coming from ad links
- technical pages and duplicates
What you should never block
A crawler needs to see a page the way a person does. If you block the folders holding your styles and scripts, Google cannot properly judge your layout or how it works on a phone, which is a real risk when most of your visitors browse on mobile.
Do not block images either if you want visitors from image search, which can be a useful source for products and services. And of course keep your main sections open: catalog, services, blog, contacts. It sounds obvious, but accidental blocks are more common than you would expect.
Crawling is not the same as indexing
The most common misunderstanding: Disallow stops a crawler from visiting a page, but it does not guarantee the page stays out of search results. If other sites or your own menu link to the blocked URL, it can still appear in results with no description.
To keep a page out of the index, use a noindex meta tag. There is a catch, though: the crawler must be able to open the page to read that tag. If you also block the page in robots.txt, noindex will never be seen. So choose the right tool for the job instead of stacking both.
Common mistakes and how to test the file
The costliest mistake is a single line: Disallow: / . It is often added during development or on a staging copy to keep the site out of search, then forgotten at launch. The result is a new website that stays invisible for weeks. Other frequent slips include blocked styles and scripts, a blocked section that should generate leads, typos in paths, and a missing Sitemap line.
Check the file after every site update, CMS change or hosting move. Google Search Console shows which version of robots.txt Google sees and which pages are blocked, and Yandex Webmaster has its own robots.txt analysis tool. Test real URLs that matter: the home page, a category, a product page and an article.
Where to start
Open your-domain.uz/robots.txt and read what it says today. If the file is missing or contains Disallow: /, that is your first task. Then compare each rule against the list of sections that should appear in search.
At markdev we usually begin a technical SEO audit with robots.txt, because one line here can cancel out the work you put into content and advertising. If you would like a specialist to review your site and say what to close and what to open, tell us about your task and we will talk it through.
FAQ
Where should the robots.txt file be located?
Strictly in the root of the domain: your-domain.uz/robots.txt. A crawler will not find it in a subfolder. Each subdomain needs its own file.
Does robots.txt protect private pages from outsiders?
No. The file is public, so anyone can see which paths you mention. Protect confidential data with passwords and server-side access settings.
What if a blocked page still shows up in search?
Allow crawling of that page in robots.txt and add a noindex meta tag to it. Once the crawler revisits and sees the tag, the page will drop out of the index.
Do I need to list the sitemap in robots.txt?
It is recommended. The Sitemap line helps crawlers find your sitemap and, through it, your important pages sooner. Also submit the sitemap in Search Console and Yandex Webmaster.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.