File does not exist:../public_html/robots.txt
What is this file? and what can I do about it?
It keeps showing up in my error log.
I tried making a blank robots.txt file and placed it in my public_html folder, but I still get this "File does not exist:../public_html/robots.txt" error.
I've tried putting a meta tag in the head section:
<META NAME="ROBOTS" CONTENT="ALL">
But Validator did not like that, and now getting errors indicating that something is trying to access folders that are not linked to my public pages.
Does this meta give robots permission to crawl all over pages not linked up to the public pages?
Is there a better meta tag for the robots to use? and not let them crawl into pages they shouldn't.
thanks for any advice,
The robots.txt file is best placed in the root of your site. Not all robots look at this, so also use a meta tag. The example you gave would allow the robot to index and follow links on the page. It should validate for html, the validator has possibly picked-up on a previous error.
W3C search engine recommendations
If another site has linked to your site, then robots may index your site regardless of your efforts to keep them out.
Last edited by Fang; 07-31-2005 at 01:56 PM.
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)