With so much talk about getting into search engines, and achieving high rankings on a particular search engine, it may seem crazy to do anything which would prevent this from happening. However, there are times when you have a web site or page(s) within a web site which you don’t want to end up in a search engine.
It might be a resource page for a professor to give his/her students additional information. Or, it might be a temporary page to provide information, such as a list of links, but one which you don’t want to include as part of your site, for fear of adversely impacting your page rank.
You may say you don’t have to worry about this because you haven’t submitted your site to any search engines. However, if someone else has put a link on their site to your site, the spiders that crawl their site will eventually end up crawling your site.
There is code you can put into your HTML which will prevent the bots from spidering your page(s). This code is put in the header of your web page(s) (meaning between theandtags), in the form of a meta tag (and you thought meta tags were for keywords only).
Here are four examples of how this code will look;
<meta name=”robots” content=”index,follow” />
<meta name=”robots” content=”noindex,follow” />
<meta name=”robots” content=”index,nofollow” />
<meta name=”robots” content=”noindex,nofollow” />
There are two parameters; “index” and “follow”. “Index” refers to whether or not you want that particular page to be indexed. “Follow” refers to whether or not you want the spider to follow the links on that page. You would decide whether you wanted the page indexed, and whether you wanted the links followed. Based on that decision, you would chose one of the above options. The final code would look something like this:
<meta name=”robots” content=”noindex,nofollow” / >
That’s all there is to it!
Hope this helps!
Feel free to contact us if you have any questions.