www.webdeveloper.com
Results 1 to 4 of 4

Thread: How to stop SE bots finding /test/ directory

  1. #1
    Join Date
    Nov 2005
    Location
    Scotland, UK
    Posts
    149

    How to stop SE bots finding /test/ directory

    Our company doesn't have a dedicated testing server so we have to use www.url.co.uk/test/ for showing test pages internally or to external partners/agencies. There's no sensitive data here but it does contain versions of sections of our site that may be out of date and we'd really rather search engines didn't find it. There are obviously no inbound links from the main site but we'd rather just be safe than sorry.

    What is the best way to stop the bots from using only this directory?

  2. #2
    Join Date
    Apr 2008
    Posts
    60
    Hi

    You need to set up a robots.txt on your webserver.

    From that file you will be able to mention to SE bots not to scrawl such or such directory

    Have a look at :
    http://www.robotstxt.org/robotstxt.html

    HTH
    G.
    My SIP Switch : Open Source VoIP project

  3. #3
    Join Date
    Nov 2005
    Location
    Scotland, UK
    Posts
    149
    Thanks gbonnet. That's what I presumed but I was hoping I'd be able to have a robots.txt file only in the test directory. Not that I have a problem with the file being in root, I'm just scared I mess up and end up blocking everything!

    I've made a file called robots.txt and within that I have:

    Code:
    User-agent: *
    Disallow: /test/
    Look OK to you?

  4. #4
    Join Date
    Apr 2008
    Posts
    60
    Hi
    It seems fine.

    It's a know technologie and a strandard so you won't mess things up.
    My SIP Switch : Open Source VoIP project

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles