There are special meta robots tags for both the Google and MSN
No Comments
Written by Robert on August 11, 2008 – 8:03 am
Create a Meta Robots Tag
Sometimes you may not actually want the search engine spiders to visit certain pages within your Web site. Although this is often not the case, especially because the goals of search-engine optimization are to increase search-engine-generated traffic, there are situations where the privacy of a particular Web page or Web site is of utmost concern.
The meta robots tag allows you to identify what pages the search engines are allowed to index in their results pages and whether or not they are allowed to follow links on those pages to other Web pages or Web sites. This is especially useful if certain sections of your Web site require payment to access. The last thing you want is search engines sending visitors directly to those locations.
You can use the meta robots tag to tell a search engine spider whether or not the Web page it visits should be indexed or if links on that page should be followed. These search-engine “robots” often need to be controlled. Depending on your Web hosting, you may not have the ability to create and add a robots.txt file. If that is the case, using the meta robots tag is the only method available to at least partially control the behavior of the search-engine spiders. The meta robots tag is located in the header of an HTML document, and its syntax is as follows:
<HEAD>
<META NAME=”robots” CONTENT=”index,follow”>
or
<META NAME=”robots”
CONTENT=”noindex,follow”>
or
<META NAME=”robots”
CONTENT=”index,nofollow”>
or
<META NAME=”robots”
CONTENT=”noindex,nofollow”>
</HEAD>
The meta robots tag includes directives for the search engine spiders. The four directives available are index, noindex, follow, and nofollow. Including index in the meta robots tag tells the spiders to index that page. Noindex tells the spiders not to index that page. Including follow tells the spiders to follow links on that page. Nofollow tells the spiders not to follow links on that page. Use only one of the four variations at a time. You can also use CONTENT= “all” in exchange of “index,follow” or CONTENT= “none” in exchange for “noindex,nofollow”.
Normally, you use these directives to block the search engine spiders from indexing a certain piece of content such as a product or piece of information that a visitor would have to purchase before accessing. You may want to consider investing in a Web hosting service that allows you to use a robots.txt file if that is the case because it provides a more robust layer of protection.
There are special meta robots tags for both the Google and MSN search-engine spiders. Generally, the normal meta robots tag should be enough to control the activity of these robots, but sometimes you may want a more granular level of control over what spiders are allowed to do. Google’s spider is called Googlebot. To control Googlebot, instead of the NAME attribute being set to robots in your meta robots tag, set it to Googlebot. You have a few extra options besides the index, noindex, follow, and nofollow directives.
You can also tell Googlebot not to archive a copy of your Web page in its cache by using the directive noarchive. If you do not want a description of your Web page to show up in the Google results page, you can use the directive nosnippet. MSN’s spider is called MSNBot. To control MSNBot, instead of setting the NAME attribute to robots in your meta robots tag, set it to MSNBot. MSNBot obeys only the noindex, nofollow, and noarchive directives.
