Results 1 to 2 of 2

Thread: Benefits of Excluding robots

  1. #1
    Join Date
    Sep 2012

    Benefits of Excluding robots

    Excluding robots from certain sections of the site can benefit you :

    1. You can prevent search engine crawlers from eating up unnecessary bandwidth of your site, For some hosting bandwidth is costly and it is important bandwidth is saved by excluding crawlers from certain sections.

    2. Excluding robots from comments section if you run a blog is very good, It prevents robots from seeing or giving credit to any unnecessary links.

    3. Excluding robots also helps in preventing sensitive information of your site not getting exposed in search results

    I just know things above, so can you list more?

  2. #2
    Join Date
    Jun 2012
    It can be used for the files or pages which owner does not want bots to crawl.
    it helps in hiding urls which were showing in crawl errors, we can put those also in robot.txt

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
HTML5 Development Center