WordPress robots.txt: What to Block and What to Leave Open

Virtual vs physical file: which one is live

If your site root has no robots.txt, WordPress generates a virtual one on the fly. Open yourdomain.uz/robots.txt in a browser and you will see a few basic rules: the admin area is blocked, while admin-ajax.php stays accessible because some themes and plugins rely on it. SEO plugins can add your sitemap address and let you edit the file from the dashboard.

If you upload a physical robots.txt to the root, it completely replaces the virtual one. From then on, changes made in a plugin do nothing. This is a classic source of confusion: the owner edits rules in one place while another file is actually being served. Always verify the result at the real /robots.txt address.

A sensible minimum for WordPress robots.txt

Every disallow line should have a clear answer to the question, why would a crawler need to go there? For most sites a short set is enough:

• The /wp-admin/ folder, with an exception for admin-ajax.php
• Internal search results, meaning URLs with the ?s= parameter
• On a WooCommerce shop: cart, checkout and the customer account area
• Utility URLs with parameters that create duplicates, if canonicals or redirects cannot solve it
• A Sitemap line pointing to your sitemap

Example: a shop that takes Click and Payme payments

Picture a Tashkent store on WooCommerce where customers pay through Click or Payme. Nobody searches Google for a cart or checkout page, so blocking those makes sense. Product pages, categories and blog articles must stay fully open.

Be careful with filters and sorting. If a filtered page such as white sneakers brings people from search, blocking it costs you traffic. Before adding a rule, check which of these pages actually receive visits.

What you should never block

Outdated templates tell you to block wp-content and wp-includes entirely. Today that is a mistake. Those folders hold styles, scripts and images, and search engines render a page much like a browser does. Without them the crawler sees bare text and may decide the page is hard to use on a phone. For a business whose customers often arrive from Instagram or Telegram on mobile, that is a real risk.

Do not block your uploads folder either, since product photos can bring visitors from image search. And avoid pasting long templates from other sites: they usually include rules for sections your site does not have.

What robots.txt cannot do, and the costliest mistake

A disallow rule limits crawling, but it does not guarantee a page leaves the index. If other pages link to a blocked URL, it can still appear in results without a description. To remove a page from search, use a noindex meta tag and keep the page crawlable, otherwise the crawler never sees the tag. For categories, tags or duplicates you want hidden, set noindex in your SEO plugin instead of blocking them here.

Old directives such as the Yandex main mirror setting are obsolete too: mirrors are now handled through redirects and Webmaster settings.

The costliest mistake is blocking the whole site. During development it is often closed with Disallow: / and the rule is forgotten at launch, so the site vanishes from search. A similar trap is the WordPress reading setting that asks search engines not to index the site. Check both after every launch or major update.

How to test the file

Testing takes a few minutes:

• Open /robots.txt in your browser and read it line by line
• Review the robots.txt report in Search Console
• Test individual URLs with the robots.txt analyzer in Yandex Webmaster
• Use URL inspection to confirm that key pages are not blocked

When to revisit the rules

Return to the file when you add a shop, change your URL structure or introduce new utility sections, and keep a copy of the working version for hosting moves. One wrong line can block crawling of the entire site, so test after every edit.

If you want a specialist to review robots.txt together with your sitemap, indexing and site structure, markdev provides SEO optimisation for websites in Uzbekistan. Tell us about your project and we will suggest where to start.