www.webdeveloper.com
Results 1 to 9 of 9

Thread: How to block all search engines not to be visible?

  1. #1
    Join Date
    Sep 2005
    Posts
    1,625

    Question How to block all search engines not to be visible?

    How to block all search engines not to be visible content with ROBOTS TXT

    Simple question...

  2. #2
    Join Date
    May 2004
    Location
    FL
    Posts
    3,447
    Are you saying you don't want any of the search engines to spider your web site? Not sure what you mean by the "not to be visible" part.
    Lee

    ""Men occasionally stumble over the truth, but most of them pick themselves up and hurry off as if nothing ever happened."" -- Sir Winston Churchill

  3. #3
    Join Date
    Sep 2005
    Posts
    1,625

    Question

    Hi,
    I would like to block Subdomain and also folders as sometimes I need testing website which is in testing period but my friends can see.

    Goal is not be seen in search results.

  4. #4
    Join Date
    May 2004
    Location
    FL
    Posts
    3,447
    You can do this:
    Code:
    User-agent: *
    Disallow: /subDomainName/
    Disallow: /testingArea/
    That will block the search engines that read the robots.txt file. (While most of them do check the file, a few spiders don't so if it is sensitive material, you should password that directory.) Meanwhile, anybody will be able to directly access the directories listed.
    Lee

    ""Men occasionally stumble over the truth, but most of them pick themselves up and hurry off as if nothing ever happened."" -- Sir Winston Churchill

  5. #5
    Join Date
    Mar 2007
    Location
    localhost
    Posts
    2,036
    As with what LeeU said... if it is sensitive then secure it.

    You do have lots of hosts that will allow you to password protect a folder but please note, some of the more "Money-Orientated" hosts will charge anything from £15 a year plus a one off fee.

    Search hard and long and you will find budget hosts that let you have access to this feature for free.

    Your alternate option is to hold these types of file outside the public_html / htdocs folder and use your server-side of choice to access it.
    Yes, I know I'm about as subtle as being hit by a bus..(\\.\ Aug08)
    Yep... I say it like I see it, even if it is like a baseball bat in the nutz... (\\.\ Aug08)
    I want to leave this world the same way I came into it, Screaming, Incontinent & No memory!
    I laughed that hard I burst my colostomy bag... (\\.\ May03)
    Life for some is like a car accident... Mine is like a motorway pile up...

    Problems with Vista? :: Getting Cryptic wid it. :: The 'C' word! :: Whois?

  6. #6
    Join Date
    Sep 2005
    Posts
    1,625
    Quote Originally Posted by LeeU View Post
    You can do this:
    Code:
    User-agent: *
    Disallow: /subDomainName/
    Disallow: /testingArea/

    Is correct:


    Disallow: /http://www.subdomain.domain.com/
    Disallow: /testing/

  7. #7
    Join Date
    May 2004
    Location
    FL
    Posts
    3,447
    No, use what I gave you above.
    Lee

    ""Men occasionally stumble over the truth, but most of them pick themselves up and hurry off as if nothing ever happened."" -- Sir Winston Churchill

  8. #8
    Join Date
    Sep 2005
    Posts
    1,625
    Quote Originally Posted by LeeU View Post
    No, use what I gave you above.
    What is meaning (technically):testingArea and what subDomainName?

  9. #9
    Join Date
    May 2004
    Location
    FL
    Posts
    3,447
    That is what you gave in the example. Just replace it with the actual ones.
    Lee

    ""Men occasionally stumble over the truth, but most of them pick themselves up and hurry off as if nothing ever happened."" -- Sir Winston Churchill

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles