I use Linux/Apache/WHM and when an account is created, you can acccess it like www.mydomain.com/~clientname. This is handy for development and waiting for DNS to update.

I'd like to block robots (okay to block everyone, maybe in .htaccess, if that makes it easier) from visiting a site like www.mydomain.com/~clientname.

Some how Google indexed pages like that for one of my clients and I need it cleaned up.