Restricting Access to Files and folders

[kdmckins]

I have a user/password section in my website. In the all the "secure" pages have validation that the user is logged in, however, I have a subdirectory that contains spreadsheets and PDF's for each user.

I learned the hard way that the folder with all of the spreadsheets and PDF can be searched and cached in HTML format (by BOTS and crawlers). It basically has no valid security.

My question:

How do you lock out those files to everyone but valid user (including BOTS and Crawlers)?

[PineSolPirate]

You probably want to turn off directory indexing. To do so, (if you are on apache) just add a .htaccess file to the subdirectory you want with the following:
IndexIgnore *

Or, if you would rather, (I'm not 100% on this one)
Options -Indexes

See: http://www.javascriptkit.com/howto/htaccess11.shtml