Benefits of Excluding robots
Excluding robots from certain sections of the site can benefit you :
1. You can prevent search engine crawlers from eating up unnecessary bandwidth of your site, For some hosting bandwidth is costly and it is important bandwidth is saved by excluding crawlers from certain sections.
2. Excluding robots from comments section if you run a blog is very good, It prevents robots from seeing or giving credit to any unnecessary links.
3. Excluding robots also helps in preventing sensitive information of your site not getting exposed in search results
I just know things above, so can you list more?
It can be used for the files or pages which owner does not want bots to crawl.
it helps in hiding urls which were showing in crawl errors, we can put those also in robot.txt
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)