Hiding your web pages from search engines

Background

Search engines and some other organisations use 'robot' or 'web crawler' software to find and index web pages and their contents. Sometimes you may want to hide a web page from a search engine, for example if you are testing a new site before launching it. There are several ways to achieve this.

Hiding your web pages

To achieve the highest level of secrecy for a digital file, simply do not put it on a web server or Internet-connected machine. This is clearly impractical for documents that you want to share with others.

Several tools can restrict access to shared documents:

• server settings for search engine access (the robots.txt file)

• the 'robots' meta tag

• server settings based on the user's IP address

• login and authentication

This article provides details about the first two options.

robots.txt file

If you have your own web server, you can create a robots.txt file that tells search engines which pages on the server they should ignore. See the Web Server Administrator's Robots Exclusion Protocol Guide for information about how to do this.

The robots exclusion protocol is supported by many of the bigger search engines, but not all of them. For information about a particular search engine's attitude to the protocol, look in the company's privacy policy.

robots meta tag

The <meta name="robots"> tag can help hide your web page from web crawlers, spiders and robots.

For complete protection, use this tag:
  <meta name="robots" content="none">

Using this metadata tag...
  <meta name="robots" content="noindex">
...will tell search engines not to index this particular page, but some may still follow links from this page to other pages and index those secondary pages.

Using this tag...
  <meta name="robots" content="nofollow">
...tells the search engines not to follow links from this particular page, but they may still index the current page.

EXTRA TIP: Make sure there are no links from other web pages to the page you want to hide. Search engine spiders/robots can (and will) follow those links to your page. Remember also that messages sent to discussion lists (for example, web-forum@unimelb) should not include the secret URL, because these discussion lists are often archived on a web site.

Related Resources

All these articles were current on 11 September 2002:

• Web Server Administrator's Robots Exclusion Protocol Guide

• Web Developers' Virtual Library entry on robots exclusion

• SearchTools.com's article on Search indexing robots and robots.txt

• Holovaty.com's weblog item on user agents and robots

• WannaBrowser.com tells you what user agent information your browser etc is providing to web servers