Re: FN-FORUM: Google crawl

date posted 16th July 2005 19:20


> How can I STOP Google searching some pages?

Two methods can be used:

1) META tag, not supported by all browsers/robots. In each page use . Also, you can use
index,nofollow etc... accordingly.

2) Exclusion protocol. This is a text file in the root directory called
robots.txt. Use the following syntax:

User-agent: *
Disallow: /
Disallow: foobar/
etc...

More info is at: http://www.robotstxt.org/.

HTH,
Emma