Why a site under development should stay out of search
A staging version is full of half-written pages, placeholder text and fake products. If it gets indexed, visitors may land on unfinished content, and Google may see the same pages on two addresses. When the real site goes live, the test copy competes with it. For an online shop, test prices or dummy items showing up publicly can also confuse customers.
Option 1: robots.txt with Disallow all
The classic rule sits in the robots.txt file at the root of the domain: “User-agent: *” followed by “Disallow: /”. The first line addresses every crawler, the second blocks every path. It takes a minute to set up, which is why developers reach for it first.
The catch is that robots.txt controls crawling, not indexing. If someone shares the staging link in a Telegram group or an email thread, a search engine can still list the bare URL without a description. And the file is a request, not a lock: well-behaved search crawlers follow it, but other bots may not.
Option 2: noindex on every page
A robots meta tag with noindex, or an X-Robots-Tag header sent by the server, lets the crawler open the page and read the instruction not to show it. This is more reliable for keeping pages out of results, but it has one strict condition: do not combine it with a Disallow rule. A blocked crawler never loads the page, so it never sees the noindex.
Option 3: password protection
The most dependable method is to put the staging domain behind HTTP authentication or an IP allowlist. Crawlers get an access-denied response and see nothing, and outsiders cannot browse your unfinished store or its test orders. Your client and team log in as usual, and the protection is easy to remove at launch. Many hosting panels offer it as a single setting.
Which one to choose
Match the method to the situation:
- Staging on a separate subdomain or domain: use a password
- Work in progress that must stay visible by link: use noindex in the global template
- A quick, temporary fix: use robots.txt, and never publish the URL anywhere
- Maximum safety: combine a password with noindex, so a removed password does not expose the site
The expensive mistake: forgetting to open the site
Most damage happens at launch. The staging files move to the live domain together with “Disallow: /”, or a CMS option that discourages search engines stays switched on. The site works, ads from Instagram or Google bring visitors, but organic search sends nobody. Owners often spot it only when they wonder why enquiries never arrive from search.
Add these checks to the launch list:
- Open robots.txt on the live domain and confirm there is no blanket block
- View the source of the homepage and a few inner pages and look for noindex
- Turn off the CMS setting that hides the site from search engines
- Remove the password and confirm pages respond normally
- Submit the sitemap in Google Search Console
- Inspect the homepage with the URL inspection tool
If the staging site is already indexed, and how to brief your developer
First close access with a password, or add noindex while keeping crawling open so the bot can read it. Then use the temporary removal tool in Search Console. If the test version lived on a subdomain, verify that subdomain as its own property. When the test copy has inbound links, a 301 redirect to the live domain is often the cleanest fix.
Prevent the problem in the brief: state where staging lives, how it is protected, and who removes the restrictions at launch. Ask for access to the staging domain yourself, so you can see it is behind a password and not just a robots.txt line. After launch, keep a copy of the live robots.txt for comparison during future updates.
If you want a launch that does not cost you search traffic, markdev can run the technical checks and help with SEO optimisation in Uzbekistan. Tell us about your project and we will talk it through.
FAQ
What does Disallow: / do in robots.txt?
Together with “User-agent: *”, it asks all crawlers not to crawl any page of the site. It does not guarantee the URL stays out of the index if other sites link to it.
Can I use robots.txt and noindex together?
Not for the same pages. If robots.txt blocks a page, the crawler never loads it and cannot read the noindex tag.
What is the safest way to hide a staging site?
A password or IP restriction. Crawlers and visitors are refused access, so protection does not depend on a bot obeying rules.
Why is my new site missing from Google after launch?
A leftover Disallow: / rule, a noindex tag in the template or a CMS visibility setting is the usual cause. Check those first.
Let’s look at your task for free
Leave your contact — in a 15-minute call we’ll show where leads get lost and what to fix first.