[RESOLVED] shared ssl / bot blocking

[TecBrat]

We have a dedicated server. (LAMP) For any client on our server that needs ssl, we can use "secure.ourdomain.net/~theirusername/..." The problem I am trying to solve is demonstrated by doing a google search like "site:secure.ourdomain.net" and it finds secure pages from many of our clients' sites.

How do I prevent following and indexing these pages?

[svidgen]

Refer to http://www.robotstxt.org/robotstxt.html

BUT, be aware that these rules need not be adhered to. And, while you'd be hard-pressed to find a major crawler that indexes pages disallowed by robots.txt, there are no legal or "mechanical" reasons that a search engine can't simply ignore the file. (none that I am aware of, anyway)

[TecBrat]

svidgen,
I am familiar with robots.txt. The problem is that I am trying to deal with this on a multiple domain level. secure.ourdomain.net/robots.txt does not exist. There is no way for me to create one. I could create secure.ourdomain.net/~username/robots.txt, but anything in that file would actually effect www.usersdomain.com. Do you understand? Do you have any other ideas?

I also understand that robots.txt file only applies to polite robots and might actually help impolite ones find sensitive data.

[criterion9]

Do you have a root level folder for "secure.ourdomain.net"? If so place your robots.txt there with a blanket deny all. That should affect all sub folders (~theirusername/) equally without affecting theiruserdomain.com.

[TecBrat]

That would have been too easy. Unfortunately, I have not found a way to access such a folder. I have been using SSH a little more lately. Maybe I'll poke around in my server and see if I can find it.

edit:
I actually checked an old ticket with my hosting company and they said there is no way to write to such a folder.

[TecBrat]

I posted another ticket to my hosting company and they finally gave me the right answer:

For https://secure.ourdomain.net/ the document root is /usr/local/apache/htdocs

So I just had to go there and write that file with pico.
Hopefully this helps someone else too. :)

[svidgen]

Glad you got it figured out.