Click to See Complete Forum and Search --> : robots.txt logs: results?


anothen
08-25-2004, 05:38 PM
I've discovered that msnbot has crawled my site. I wanted to know the results, I don't understand it too well, is the "...404.. " an error code?:
65.54.188.66 - - [25/Aug/2004:13:03:52 -0400] "GET /robots.txt HTTP/1.0" 404 640 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
65.54.188.66 - - [25/Aug/2004:13:03:53 -0400] "GET /javascript/inputwrite.html HTTP/1.0" 404 640 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"

The IP I understand, the time stamp I understand, what does "...

"GET /javascript/inputwrite.html HTTP/1.0" 404 640 "-" "msnbot/0.11

..." mean?
and obviously the (+http://search.msn.com/msnbot.htm) is the crawler.
In any case, this means that a crawler is trying to index my site?
Also, do I need to have the robots.txt file?

hdtvtechno
09-13-2004, 02:17 PM
yes u need to have the robots.txt


http://www.hdtvtechno.com

hdtvtechno
09-22-2004, 05:31 AM
it looks like the msnbot got a 404 error when reached ur website.

Robert Wellock
09-29-2004, 01:46 PM
Most likely a JavaScript link it couldn't follow hence 404.