www.webdeveloper.com
Results 1 to 7 of 7

Thread: ROBOTS.TXT is not hiding files

  1. #1
    Join Date
    Jun 2005
    Posts
    48

    None

    NONE
    Last edited by rwhite; 07-25-2005 at 02:42 AM.

  2. #2
    Join Date
    Apr 2003
    Location
    Netherlands
    Posts
    21,654
    Not all bots read the robots.txt file. Another site can have links to your files that the bots follow.
    Try adding these to your documents:
    Code:
    <meta name="robots" content="noindex">
    <meta name="robots" content="nofollow">
    If you don't want your documents indexed don't put them on internet.

  3. #3
    Join Date
    Jun 2005
    Posts
    48

    None

    NONE
    Last edited by rwhite; 07-25-2005 at 02:41 AM.

  4. #4
    Join Date
    Apr 2003
    Location
    Netherlands
    Posts
    21,654
    Moving them to a different directory will help, but eventually they may be found again by less discerning bots.

  5. #5
    Join Date
    Nov 2002
    Location
    XYZZY - UK
    Posts
    1,760
    Use .htaccess if you can to make the folders private or stop external sites linking via ModRewrite.

  6. #6
    Join Date
    Jun 2005
    Posts
    48

    None

    NONE
    Last edited by rwhite; 07-25-2005 at 02:41 AM.

  7. #7
    Join Date
    Nov 2002
    Location
    XYZZY - UK
    Posts
    1,760

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles