If you have site running in FRAMES you might not want the search engines to list pages INSIDE FRAMESETS without thier own navigation
Similarly if you set up HTML to go into frames as lists of links or headers & footers you might not want them listed
AND if a page is a list of links but of no intrinsic value you might want it NOINDEX but allow FOLLOW
As rightly said robots.txt can be used to explicitly exclude certain file types or directories
search robots.txt at google for more info:
There is no guarantee that ALL search engines will abey these rules THE RULE IS ,... if you don't want ANYONE to find it DON'T put it on any webpages unless you believe you can keep it safe with non java .htaccess password control !