Why Bitrix needs a careful robots.txt
Crawlers give every site a limited amount of attention. If they keep finding duplicate and service URLs, they reach the pages that actually bring in orders later. For a store with a catalog and filters, such as an electronics shop selling on its own site alongside Uzum or OLX, that delay is easy to notice.
The Bitrix search optimization module includes a robots.txt editor with ready-made rules, and you can also edit the file through your hosting. The presets are a useful starting point, not a finished answer. Your sections, template and components are specific to you, so read each rule and understand it before accepting it.
What to block
Most Bitrix sites block three groups of addresses: technical folders, user pages with no search value, and parameter-based duplicates.
- technical files inside /bitrix/, but not the files needed to render pages
- service locations such as /local/ and /cgi-bin/, plus any leftover test sections
- cart and checkout, including pages shown after a Click or Payme payment
- account area, login, registration and password recovery
- product comparison and wishlist
- internal site search results
- print versions of pages
What you must not block
Google judges a page the way a visitor sees it in a browser. If stylesheets and scripts are blocked, the crawler may see a broken layout and treat the page as hard to use on mobile. Bitrix templates usually live in /bitrix/templates/ or /local/templates/, so blocking a whole folder is risky. Keep CSS and JS accessible.
Leave product images open too if image search matters to you, and check that a rule on /upload/ does not hide your catalog photos. The most dangerous mistake is a leftover Disallow: / after moving the site from a test server to production.
URL parameters, pagination and Clean-param
Sorting, view mode, items per page, session and print parameters each create a new URL with the same content. You can block them with Disallow rules using the asterisk wildcard. Write the pattern carefully, because a rule that is too broad will also hide pages you want indexed. Filter pages you promote deliberately, such as sofas in Tashkent, should stay open with their own clean address.
Yandex supports the Clean-param directive, which tells the robot that a parameter does not change the content so it can merge those URLs with the main one. It is gentler than a full block and works well for ad tags and sorting. Google ignores Clean-param, so for Google you handle duplicates with canonical tags.
Pagination is debatable. If some products can only be reached through paginated pages, blocking them makes discovery harder. Often it is better to keep pagination open and rely on category structure and the sitemap.
Blocking does not remove a page from search
robots.txt stops crawling, not indexing. If a page is already indexed and has links pointing to it, it can stay in results, often with no description, because the robot no longer visits to see changes.
To remove a page, keep it crawlable and add a noindex meta tag. Once it drops out of the index, you can add a robots.txt block if you still need one.
Multiple sites and the Sitemap line
If one Bitrix installation serves a Russian site and an Uzbek version on a subdomain, the file may be shared. Make sure each domain returns its own rules and its own Sitemap line, not the neighbor's map. Put the full sitemap URL at the end of the file, and if there are several maps, point to the index file.
How to test before you save
Open yourdomain/robots.txt and confirm it returns status 200. Then paste the rules into the robots.txt analysis tool in Yandex Webmaster and check a list of URLs: home page, categories, product cards and articles should be allowed, while the cart, account and sorted URLs should be blocked. After publishing, review the robots.txt report in Search Console.
If you are unsure about your current file, markdev can review it as part of SEO optimisation in Uzbekistan and tell you what to fix. Send us your task and we will go through it together.
FAQ
Where do I edit robots.txt in Bitrix?
Either through your hosting or in the editor of the search optimization module. Always compare the preset rules with your own site structure.
Can I block the whole /bitrix/ folder?
No. Template styles and scripts are needed to render pages. Block technical subfolders and leave CSS, JS and images accessible.
Should I block pagination?
Usually not. If some products are reachable only through paginated pages, blocking them makes it harder for crawlers to find those products.
My page is blocked in robots.txt but still in search. Why?
A block does not remove a page from the index. Open it for crawling, add noindex, wait for it to drop out, then block it if needed.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.