Robots.txt is a text tell search engine to avoid crawling of secret pages like the personal images folder, website administration folder, customerís test folder of a web developer ect.