dcsimg
www.webdeveloper.com
Page 1 of 3 123 LastLast
Results 1 to 15 of 31

Thread: what is the role of robot.txt in SEO?

  1. #1

    what is the role of robot.txt in SEO?

    Hello Guys,
    Please share your review about robot.txt, what is this and how its important for seo. I am not more familiar to seo task so please help me to know about it. Thanks
    Share on Google+

  2. #2
    Join Date
    Jan 2013
    Posts
    52
    Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.
    Share on Google+

  3. #3
    Join Date
    Mar 2013
    Posts
    139
    You can make use of robot.txt files if you don't want to crawl any page etc..
    Share on Google+

  4. #4
    Join Date
    Mar 2013
    Location
    USA
    Posts
    3
    It is important part for the site . From that you can restrict the crawler to carwl that particular page you doesn't want to crawl.
    Share on Google+

  5. #5
    Join Date
    Nov 2011
    Location
    chennai
    Posts
    73
    Robots.txt helps, not to index website personal pages in search engines. It acts like security for your personal page.
    Share on Google+

  6. #6
    Join Date
    Feb 2013
    Location
    Mississauga, Canada
    Posts
    152
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na´ve to rely on robots.txt to protect it from being indexed and displayed in search results.
    *** spam signature deleted by moderator - please do not put it back ***
    Share on Google+

  7. #7
    Join Date
    Apr 2013
    Posts
    9
    way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots metatag.
    Share on Google+

  8. #8
    Join Date
    Sep 2012
    Posts
    720
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
    Share on Google+

  9. #9
    Join Date
    Feb 2012
    Posts
    83
    It is used to restrict or allow bots to visit our site.
    Share on Google+

  10. #10
    Join Date
    Nov 2014
    Location
    Bangalore
    Posts
    30
    We should use robot.txt if we want to restrict crawlers to crawl any private pages. You can do it using notepad file and upload it in your server. robots.txt tells google, yahoo, bing crawlers what not to crawl on your pages.

    An example of how we use it is the following.

    We have a test directory on our server where we upload sites to be tested before we go public. Obviously, we do not want the crawlers to take note of this directory since it is a directory for testing only. Below is the robots.txt file contents.


    User-agent: *

    Disallow:/testing/
    Share on Google+

  11. #11
    Join Date
    Apr 2015
    Location
    Pune
    Posts
    21
    To have Robots.txt for your websites is one oh the important factor fro SEO perspective.
    It is a text file which tell serach engine about which pages of your website should be crawled and which should not...
    It is simple text file which is uploaded on server.
    Share on Google+

  12. #12
    Join Date
    Dec 2014
    Location
    Chennai
    Posts
    101
    Quote Originally Posted by iTech View Post
    You can make use of robot.txt files if you don't want to crawl any page etc..
    Quote Originally Posted by dennyleon View Post
    Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.
    Most of the Search engines won't fllow the Robots.txt rules. but the purpose of Robots.txt is to control or specify the bots what should they do.

    Kind Regards
    Share on Google+

  13. #13
    Join Date
    Feb 2015
    Posts
    79
    Robots.txt is stop the search engines crawling your web site or web page. You can show your site online but without crawling and it is use when you are working on your site or page without showing anyone.
    Share on Google+

  14. #14
    Join Date
    May 2015
    Location
    noida
    Posts
    1

    What is Robot.txt

    how to crawl and index pages on their website.

    www.kampuzz.com
    Share on Google+

  15. #15
    Join Date
    Dec 2014
    Posts
    142
    In a website that the site owner is asking the search engines to "skip" (or "disallow") some URL's which are not necessary.

    Syntax: User-agent:
    * Disallow: /cgi-bin/
    Disallow: /tmp/
    Disallow: /junk/
    Share on Google+

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles