What noindex actually does
Search engines work in two steps: they crawl a page, then decide whether to add it to the index. Robots.txt influences the first step by telling a crawler where not to go. Noindex influences the second: the crawler reads the page, but the page is left out of search results.
That makes noindex a page-level tool. You do not need to edit the sitewide robots.txt file, which is handy when your site runs on a builder or a CMS where that file is awkward to change.
Which pages are worth hiding
A useful test: would I want someone to land on this page straight from search? If not, but the site still needs it, noindex is a good fit. Take an online shop that accepts payment through Click or Payme. Its order confirmation page helps the buyer, yet in search results it only confuses people.
- internal site search results
- cart, checkout and customer account pages
- thank-you pages after a form or payment
- print and comparison versions of pages
- test or draft pages published by accident
- thin tag pages that duplicate your categories
Two ways to add the directive
The first is a meta tag inside the head of the page: name="robots" with content="noindex". The second is the X-Robots-Tag HTTP header, sent by the server with the response. The header is the only option for files without HTML, for example a PDF price list or internal manual that has surfaced in search.
If you need to address one search engine only, replace robots in the name attribute with googlebot or yandex. In practice most service pages are hidden from every engine at once.
Adding nofollow to such pages is usually unnecessary. A page with plain noindex still lets the crawler follow its links and discover other sections of the site.
The classic trap: noindex plus Disallow
A very common mistake is to block a page in robots.txt and also add noindex to it. A crawler that is not allowed to fetch the page never sees the tag. If other sites link to that URL, it can stay in the index even though its content was never read.
The safe order is: add noindex and keep the page crawlable, wait until it drops out of the index, and only then, if you really need to save crawl resources, block the section in robots.txt. Also remember that Google ignores a noindex line written inside robots.txt itself.
Mistakes that quietly cost you traffic
The most expensive case is noindex landing in a template, because it then spreads across hundreds of pages at once. Watch for these:
- a tag left in the template after development
- an SEO plugin update that adds noindex to categories or products
- a CMS setting such as “discourage search engines from indexing this site”
- a noindex page listed in the sitemap
- a canonical link pointing to a noindex page
- two conflicting robots meta tags on one page
How to check that it works
Open the page source and look for the robots meta tag in the head; make sure there is only one. For the X-Robots-Tag header, check the server response in your browser’s developer tools. Then run the URL through the inspection tool in Google Search Console and check the page status in Yandex Webmaster to see whether the directive is detected.
Pages do not vanish instantly, because the crawler has to revisit them. Requesting a recrawl speeds things up. After every site release, it is worth a quick look at your main service and category pages, since that is where an accidental noindex hurts most.
When noindex is the wrong tool
If a page is gone for good and has a replacement, use a 301 redirect. If it duplicates another page, use a canonical link. If the content is private, protect it with a password, because noindex hides a page from search but not from anyone holding the link.
If you are unsure which pages to hide and which to promote, that is a good question for an SEO audit. markdev works on SEO optimisation for websites in Uzbekistan, and we are happy to look at your site and talk through the task.
FAQ
How long does it take for a page to disappear after adding noindex?
There is no fixed time. The page leaves the index once the crawler revisits it and reads the directive. A recrawl request in Search Console or Yandex Webmaster can speed this up.
Can I put noindex in robots.txt?
No. Google does not support a noindex rule inside robots.txt, so it has no effect. Place the directive on the page itself or in the HTTP header.
Should I use noindex, nofollow on a cart page?
Usually plain noindex is enough. Nofollow turns the page into a dead end for the crawler, while its links can still help it find other sections.
How do I hide a PDF from search results?
A PDF has no head section, so use the X-Robots-Tag: noindex HTTP header. It is configured on the server, for example in nginx or Apache.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.