www.webdeveloper.com
Results 1 to 4 of 4

Thread: robots.txt logs: results?

Hybrid View

  1. #1
    Join Date
    Apr 2003
    Posts
    136

    robots.txt logs: results?

    I've discovered that msnbot has crawled my site. I wanted to know the results, I don't understand it too well, is the "...404.. " an error code?:
    65.54.188.66 - - [25/Aug/2004:13:03:52 -0400] "GET /robots.txt HTTP/1.0" 404 640 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
    65.54.188.66 - - [25/Aug/2004:13:03:53 -0400] "GET /javascript/inputwrite.html HTTP/1.0" 404 640 "-" "msnbot/0.11 (+http://search.msn.com/msnbot.htm)"
    The IP I understand, the time stamp I understand, what does "...

    "GET /javascript/inputwrite.html HTTP/1.0" 404 640 "-" "msnbot/0.11

    ..." mean?
    and obviously the (+http://search.msn.com/msnbot.htm) is the crawler.
    In any case, this means that a crawler is trying to index my site?
    Also, do I need to have the robots.txt file?

  2. #2
    yes u need to have the robots.txt


    http://www.hdtvtechno.com
    Last edited by hdtvtechno; 09-16-2005 at 09:35 AM.

  3. #3
    it looks like the msnbot got a 404 error when reached ur website.

  4. #4
    Join Date
    Nov 2002
    Location
    XYZZY - UK
    Posts
    1,760
    Most likely a JavaScript link it couldn't follow hence 404.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles