PDA

Click to See Complete Forum and Search --> : supressing search engine spiders

laria
11-20-2002, 10:33 PM
hi everybody,

is there a way to prevent being listed by search engines?

thanks :)
Zach Elfers
11-20-2002, 11:27 PM
Yes there is. In the <head> section of your page, type:

<meta name="robots" content="noindex, nofollow">

noindex means that the search engine robots will NOT index your page. nofollow means that they will not follow the links on your page.
lcole
11-20-2002, 11:28 PM
(Taken from 10 ways not to promote your website (http://webdevelopersjournal.com/articles/10_ways_not_to_promote_your_web_site.html))

Create a text file called "robots.txt," and place it in your Web site's root directory (usually the "htdocs" directory). This file has a list of pages or directories that you want to keep the spiders out of, and it looks something like this:

User-agent: *
Disallow: /test/
Disallow: /temporary/
Disallow: /templates/

This tells all visiting spiders not to fool with any of the 3 named directories. Note that the directory names must end with a "/".


You can find further info at
Robots Exclusion (http://www.robotstxt.org/wc/exclusion.html)
teeka
11-21-2002, 04:02 PM
Hello everybody!! Just signed on.....

Also just published my first web site and, of course, am full of questions!


Concerning this:

<meta name="robots" content="noindex, nofollow">

why woud you not want to be listed by search engines? Isn't the whole idea of it is for people to be able to find your page?

Is it not a good thing that robots index a page or follow links?
Can they mess up my pages somehow?

Thanks
:D
cxema
11-21-2002, 05:43 PM
Here I give you a link to an old article (october 2001) that is most explanatory on the robots topic.
Hope you enjoy it
http://www.searchenginewatch.com/searchday/01/sd1024-robots.html
Zach Elfers
11-21-2002, 11:12 PM
Originally posted by teeka
Hello everybody!! Just signed on.....

Also just published my first web site and, of course, am full of questions!


Concerning this:

<meta name="robots" content="noindex, nofollow">

why woud you not want to be listed by search engines? Isn't the whole idea of it is for people to be able to find your page?

Is it not a good thing that robots index a page or follow links?
Can they mess up my pages somehow?

Thanks
:D


Sometimes you just don't want certain pages listed.
teeka
11-22-2002, 11:22 AM
Thanks cxema....lots of reading for me now!!
Dr.5150
11-22-2002, 01:46 PM
I don't like them indexing some of my pages.

ie.
Pages setup for private use. (My MP3 pages that change dynamically :D )

Not alowing them to index the individual pages of a site other than the home page.
johnbullas
11-23-2002, 09:38 AM
If you have site running in FRAMES you might not want the search engines to list pages INSIDE FRAMESETS without thier own navigation

Similarly if you set up HTML to go into frames as lists of links or headers & footers you might not want them listed

AND if a page is a list of links but of no intrinsic value you might want it NOINDEX but allow FOLLOW

As rightly said robots.txt can be used to explicitly exclude certain file types or directories

search robots.txt at google for more info:

http://www.robotstxt.org/wc/norobots.html

http://www.robotstxt.org/wc/meta-user.html

There is no guarantee that ALL search engines will abey these rules THE RULE IS ,... if you don't want ANYONE to find it DON'T put it on any webpages unless you believe you can keep it safe with non java .htaccess password control ! ;)

John
teeka
11-26-2002, 12:48 PM
Thanks to all...I appreciate all the input. Being as this is my first web page and it is pretty simple, I don't think I have to worry about this right now, but good to know for the future!

As an aside...great forum! I've learned more from this site alone than I have in multiple other sites combined!

Thanks...