robots.txt logs: results?
I've discovered that msnbot has crawled my site. I wanted to know the results, I don't understand it too well, is the "...404.. " an error code?:
The IP I understand, the time stamp I understand, what does "...
"GET /javascript/inputwrite.html HTTP/1.0" 404 640 "-" "msnbot/0.11
..." mean?
and obviously the (+http://search.msn.com/msnbot.htm ) is the crawler.
In any case, this means that a crawler is trying to index my site?
Also, do I need to have the robots.txt file?
yes u need to have the robots.txt
http://www.hdtvtechno.com
Last edited by hdtvtechno; 09-16-2005 at 09:35 AM .
it looks like the msnbot got a 404 error when reached ur website.
Most likely a JavaScript link it couldn't follow hence 404.
Thread Information
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)
Posting Permissions
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
Forum Rules
Bookmarks