[OT] FN-FORUM -Robots.txt

date posted 23rd December 2002 09:19

Bob Dowler
I-Net Consulting Ltd
[EMAIL REMOVED]
www.i-net-consulting.co.uk

Just been checking my web site stats and noticed something strange...
I had some failure codes reported, code 404 Not found. When I dug a little
deeper, I found some stats that told me the URL's that were being looked for
when this error was reported...
Apparently they were '/robots.txt' and '/favicon.ico'
Neither of these are on my website so why are they being looked for?
Is this a crawler hit that looked for something to rank the site by that was
missing?
Confused,
Bob