Robots.txt is a text tell search engine to avoid crawling of secret pages like the personal images folder, website administration folder, customer’s test folder of a web developer ect.