dcsimg
www.webdeveloper.com
Results 1 to 12 of 12

Thread: what is the role of robot.txt in SEO?

  1. #1

    what is the role of robot.txt in SEO?

    Hello Guys,
    Please share your review about robot.txt, what is this and how its important for seo. I am not more familiar to seo task so please help me to know about it. Thanks

  2. #2
    Join Date
    Jan 2013
    Posts
    52
    Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.

  3. #3
    Join Date
    Mar 2013
    Posts
    139
    You can make use of robot.txt files if you don't want to crawl any page etc..

  4. #4
    Join Date
    Mar 2013
    Location
    USA
    Posts
    3
    It is important part for the site . From that you can restrict the crawler to carwl that particular page you doesn't want to crawl.

  5. #5
    Join Date
    Nov 2011
    Location
    chennai
    Posts
    73
    Robots.txt helps, not to index website personal pages in search engines. It acts like security for your personal page.

  6. #6
    Join Date
    Feb 2013
    Location
    Mississauga, Canada
    Posts
    152
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na´ve to rely on robots.txt to protect it from being indexed and displayed in search results.
    *** spam signature deleted by moderator - please do not put it back ***

  7. #7
    Join Date
    Apr 2013
    Posts
    9
    way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots metatag.

  8. #8
    Join Date
    Sep 2012
    Posts
    720
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.

  9. #9
    Join Date
    Feb 2012
    Posts
    83
    It is used to restrict or allow bots to visit our site.

  10. #10
    Join Date
    Nov 2014
    Location
    Bangalore
    Posts
    24
    We should use robot.txt if we want to restrict crawlers to crawl any private pages. You can do it using notepad file and upload it in your server. robots.txt tells google, yahoo, bing crawlers what not to crawl on your pages.

    An example of how we use it is the following.

    We have a test directory on our server where we upload sites to be tested before we go public. Obviously, we do not want the crawlers to take note of this directory since it is a directory for testing only. Below is the robots.txt file contents.


    User-agent: *

    Disallow:/testing/

  11. #11
    Join Date
    Apr 2015
    Location
    Pune
    Posts
    11
    To have Robots.txt for your websites is one oh the important factor fro SEO perspective.
    It is a text file which tell serach engine about which pages of your website should be crawled and which should not...
    It is simple text file which is uploaded on server.

  12. #12
    Join Date
    Dec 2014
    Location
    Chennai
    Posts
    70
    Quote Originally Posted by iTech View Post
    You can make use of robot.txt files if you don't want to crawl any page etc..
    Quote Originally Posted by dennyleon View Post
    Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.
    Most of the Search engines won't fllow the Robots.txt rules. but the purpose of Robots.txt is to control or specify the bots what should they do.

    Kind Regards

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles