www.webdeveloper.com
Results 1 to 2 of 2

Thread: What Is Robot.txt?

Hybrid View

  1. #1
    Join Date
    Sep 2012
    Posts
    720

    What Is Robot.txt?

    Robots.txt is a text tell search engine to avoid crawling of secret pages like the personal images folder, website administration folder, customerís test folder of a web developer ect.

  2. #2
    Join Date
    Jul 2012
    Posts
    18
    Robot.txt is way to instruct the search engine crawlers that what to crawl and what to not because there are so many file which useless foe search engines and we don't want to let them indexed. If you want to prevent the spiders to index a specific file or folder on the website then you can easily disallow the spiders to index them. Following is the syntax of the robot.txt:-
    User-agent: *
    Disallow: /
    Where ‘User-agent: *’ defines that it is applicable for all bots of search engines and ‘Disallow: /’ tells search engine bots that they should not visit any page on the website.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles