What I am talking about is “robots.txt”. This is a text file that is uploaded to the root of your website and works with the search engines.
Now, you don’t have to have Robots on your website; but not having this file may cause some issues. Let’s say you have duplicate pages in your site – one for viewing and the other one for printing. You may not want the search engines to index both pages because this could impose a penalty for duplicate content. You also probably wouldn’t want your print page to be indexed, as it would have nothing referring back to your website on it.
A robots.txt file tells the search engines the pages that you don’t want them to visit on your website. For the most part the good search engines will obey what you ask them to do. There are some rogue search engines out there that will bypass the robots.txt file, so if you have some sensitive data on your website, you need to lock them up.