www.webdeveloper.com
Results 1 to 2 of 2

Thread: File does not exist:../public_html/robots.txt

  1. #1
    Join Date
    Jul 2005
    Location
    Colorado
    Posts
    20

    File does not exist:../public_html/robots.txt

    What is this file? and what can I do about it?
    It keeps showing up in my error log.
    I tried making a blank robots.txt file and placed it in my public_html folder, but I still get this "File does not exist:../public_html/robots.txt" error.

    I've tried putting a meta tag in the head section:
    <META NAME="ROBOTS" CONTENT="ALL">

    But Validator did not like that, and now getting errors indicating that something is trying to access folders that are not linked to my public pages.

    Does this meta give robots permission to crawl all over pages not linked up to the public pages?

    Is there a better meta tag for the robots to use? and not let them crawl into pages they shouldn't.

    thanks for any advice,
    René
    René O'Deay, author, photojournalist, Mac Addict

    iMac G3 Mac OS X 10.3.6

    www.reneodeay.com

  2. #2
    Join Date
    Apr 2003
    Location
    Netherlands
    Posts
    21,654
    The robots.txt file is best placed in the root of your site. Not all robots look at this, so also use a meta tag. The example you gave would allow the robot to index and follow links on the page. It should validate for html, the validator has possibly picked-up on a previous error.
    robot faq
    W3C search engine recommendations
    If another site has linked to your site, then robots may index your site regardless of your efforts to keep them out.
    Last edited by Fang; 07-31-2005 at 12:56 PM.
    At least 98% of internet users' DNA is identical to that of chimpanzees

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles