dcsimg
www.webdeveloper.com
Results 1 to 10 of 10

Thread: supressing search engine spiders

  1. #1
    Join Date
    Nov 2002
    Location
    Canada
    Posts
    11

    Question supressing search engine spiders

    hi everybody,

    is there a way to prevent being listed by search engines?

    thanks

  2. #2
    Join Date
    Nov 2002
    Posts
    631
    Yes there is. In the <head> section of your page, type:

    <meta name="robots" content="noindex, nofollow">

    noindex means that the search engine robots will NOT index your page. nofollow means that they will not follow the links on your page.

  3. #3
    Join Date
    Nov 2002
    Posts
    18
    (Taken from 10 ways not to promote your website)

    Create a text file called "robots.txt," and place it in your Web site's root directory (usually the "htdocs" directory). This file has a list of pages or directories that you want to keep the spiders out of, and it looks something like this:

    User-agent: *
    Disallow: /test/
    Disallow: /temporary/
    Disallow: /templates/

    This tells all visiting spiders not to fool with any of the 3 named directories. Note that the directory names must end with a "/".


    You can find further info at
    Robots Exclusion

  4. #4
    Join Date
    Nov 2002
    Location
    Ontario, Canada
    Posts
    10

    Question

    Hello everybody!! Just signed on.....

    Also just published my first web site and, of course, am full of questions!


    Concerning this:

    <meta name="robots" content="noindex, nofollow">

    why woud you not want to be listed by search engines? Isn't the whole idea of it is for people to be able to find your page?

    Is it not a good thing that robots index a page or follow links?
    Can they mess up my pages somehow?

    Thanks

  5. #5
    Join Date
    Nov 2002
    Posts
    15
    Here I give you a link to an old article (october 2001) that is most explanatory on the robots topic.
    Hope you enjoy it
    http://www.searchenginewatch.com/sea...24-robots.html

  6. #6
    Join Date
    Nov 2002
    Posts
    631
    Originally posted by teeka
    Hello everybody!! Just signed on.....

    Also just published my first web site and, of course, am full of questions!


    Concerning this:

    <meta name="robots" content="noindex, nofollow">

    why woud you not want to be listed by search engines? Isn't the whole idea of it is for people to be able to find your page?

    Is it not a good thing that robots index a page or follow links?
    Can they mess up my pages somehow?

    Thanks

    Sometimes you just don't want certain pages listed.

  7. #7
    Join Date
    Nov 2002
    Location
    Ontario, Canada
    Posts
    10
    Thanks cxema....lots of reading for me now!!

  8. #8
    Join Date
    Nov 2002
    Posts
    15

    Robots not indexing is good sometimes

    I don't like them indexing some of my pages.

    ie.
    Pages setup for private use. (My MP3 pages that change dynamically )

    Not alowing them to index the individual pages of a site other than the home page.

  9. #9
    Join Date
    Nov 2002
    Location
    South Coast UK
    Posts
    2

    Why use NOINDEX or NO FOLLOW

    If you have site running in FRAMES you might not want the search engines to list pages INSIDE FRAMESETS without thier own navigation

    Similarly if you set up HTML to go into frames as lists of links or headers & footers you might not want them listed

    AND if a page is a list of links but of no intrinsic value you might want it NOINDEX but allow FOLLOW

    As rightly said robots.txt can be used to explicitly exclude certain file types or directories

    search robots.txt at google for more info:

    http://www.robotstxt.org/wc/norobots.html

    http://www.robotstxt.org/wc/meta-user.html

    There is no guarantee that ALL search engines will abey these rules THE RULE IS ,... if you don't want ANYONE to find it DON'T put it on any webpages unless you believe you can keep it safe with non java .htaccess password control !

    John

  10. #10
    Join Date
    Nov 2002
    Location
    Ontario, Canada
    Posts
    10
    Thanks to all...I appreciate all the input. Being as this is my first web page and it is pretty simple, I don't think I have to worry about this right now, but good to know for the future!

    As an aside...great forum! I've learned more from this site alone than I have in multiple other sites combined!

    Thanks...

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles