Re: [OT] FN-FORUM -Robots.txt

date posted 23rd December 2002 09:45

Robots.txt is the file that tells crawlers what they can and can't index.
Favicon.ico is the custom icon that IE thinks you should have - you get the
error when someone bookmarks your page with IE.

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Kathy
http://www.vendetta.co.uk
DNRC Minister for Useful but Irritating Information and Trivia
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
----- Original Message -----
From: "Bob Dowler" [EMAIL REMOVED]
To: [EMAIL REMOVED]
Sent: Monday, December 23, 2002 9:19 AM
Subject: [OT] FN-FORUM -Robots.txt


>
>
> Bob Dowler
> I-Net Consulting Ltd
> [EMAIL REMOVED]
> www.i-net-consulting.co.uk
>
> Just been checking my web site stats and noticed something strange...
> I had some failure codes reported, code 404 Not found. When I dug a little
> deeper, I found some stats that told me the URL's that were being looked
for
> when this error was reported...
> Apparently they were '/robots.txt' and '/favicon.ico'
> Neither of these are on my website so why are they being looked for?
> Is this a crawler hit that looked for something to rank the site by that
was
> missing?
> Confused,
> Bob
>
>
>
>
>
>
>
> ========
> Advertise with Freelancers.net +Additions+
> http://www.freelancers.net/advert.php
>
>
> Freelancers and Freelance Jobs:
> http://www.freelancers.net
>
> Forum FAQs:
> http://freelancers.netrickery.com
>
> To unsubscribe please email:
> [EMAIL REMOVED]
>
> If you have difficulties unsubscribing please email:
> [EMAIL REMOVED]
>