Page 1 of 3 123 LastLast
Results 1 to 15 of 31

Thread: what is the role of robot.txt in SEO?

  1. #1

    what is the role of robot.txt in SEO?

    Hello Guys,
    Please share your review about robot.txt, what is this and how its important for seo. I am not more familiar to seo task so please help me to know about it. Thanks
    Share on Google+

  2. #2
    Join Date
    Jan 2013
    Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.
    Share on Google+

  3. #3
    Join Date
    Mar 2013
    You can make use of robot.txt files if you don't want to crawl any page etc..
    Share on Google+

  4. #4
    Join Date
    Mar 2013
    It is important part for the site . From that you can restrict the crawler to carwl that particular page you doesn't want to crawl.
    Share on Google+

  5. #5
    Join Date
    Nov 2011
    Robots.txt helps, not to index website personal pages in search engines. It acts like security for your personal page.
    Share on Google+

  6. #6
    Join Date
    Feb 2013
    Mississauga, Canada
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site (i.e. it is not a firewall, or a kind of password protection) and the fact that you put a robots.txt file is something like putting a note “Please, do not enter” on an unlocked door – e.g. you cannot prevent thieves from coming in but the good guys will not open to door and enter. That is why we say that if you have really sen sitive data, it is too na´ve to rely on robots.txt to protect it from being indexed and displayed in search results.
    *** spam signature deleted by moderator - please do not put it back ***
    Share on Google+

  7. #7
    Join Date
    Apr 2013
    way to tell search engines which files and folders on your Web site to avoid is with the use of the Robots metatag.
    Share on Google+

  8. #8
    Join Date
    Sep 2012
    Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit.
    Share on Google+

  9. #9
    Join Date
    Feb 2012
    It is used to restrict or allow bots to visit our site.
    Share on Google+

  10. #10
    Join Date
    Nov 2014
    We should use robot.txt if we want to restrict crawlers to crawl any private pages. You can do it using notepad file and upload it in your server. robots.txt tells google, yahoo, bing crawlers what not to crawl on your pages.

    An example of how we use it is the following.

    We have a test directory on our server where we upload sites to be tested before we go public. Obviously, we do not want the crawlers to take note of this directory since it is a directory for testing only. Below is the robots.txt file contents.

    User-agent: *

    Share on Google+

  11. #11
    Join Date
    Apr 2015
    To have Robots.txt for your websites is one oh the important factor fro SEO perspective.
    It is a text file which tell serach engine about which pages of your website should be crawled and which should not...
    It is simple text file which is uploaded on server.
    Share on Google+

  12. #12
    Join Date
    Dec 2014
    Quote Originally Posted by iTech View Post
    You can make use of robot.txt files if you don't want to crawl any page etc..
    Quote Originally Posted by dennyleon View Post
    Robot text file is an essential part of site from crawlers point of view you may save all those files in it taht you dont want to read by crawlers.
    Most of the Search engines won't fllow the Robots.txt rules. but the purpose of Robots.txt is to control or specify the bots what should they do.

    Kind Regards
    Share on Google+

  13. #13
    Join Date
    Feb 2015
    Robots.txt is stop the search engines crawling your web site or web page. You can show your site online but without crawling and it is use when you are working on your site or page without showing anyone.
    Share on Google+

  14. #14
    Join Date
    May 2015

    What is Robot.txt

    how to crawl and index pages on their website.

    Share on Google+

  15. #15
    In a website that the site owner is asking the search engines to "skip" (or "disallow") some URL's which are not necessary.

    Syntax: User-agent:
    * Disallow: /cgi-bin/
    Disallow: /tmp/
    Disallow: /junk/
    Share on Google+

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
HTML5 Development Center



X vBulletin 4.2.2 Debug Information

  • Page Generation 0.13039 seconds
  • Memory Usage 2,996KB
  • Queries Executed 15 (?)
More Information
Template Usage (36):
  • (1)ad_footer_end
  • (1)ad_footer_start
  • (1)ad_global_above_footer
  • (1)ad_global_below_navbar
  • (1)ad_global_header1
  • (1)ad_global_header2
  • (1)ad_navbar_below
  • (1)ad_showthread_firstpost_sig
  • (1)ad_showthread_firstpost_start
  • (1)ad_thread_first_post_content
  • (1)ad_thread_last_post_content
  • (2)bbcode_quote
  • (1)footer
  • (1)forumjump
  • (1)forumrules
  • (1)gobutton
  • (1)header
  • (1)headinclude
  • (1)headinclude_bottom
  • (15)memberaction_dropdown
  • (1)navbar
  • (4)navbar_link
  • (1)navbar_moderation
  • (1)navbar_noticebit
  • (1)navbar_tabs
  • (2)option
  • (1)pagenav
  • (1)pagenav_curpage
  • (2)pagenav_pagelink
  • (15)postbit
  • (15)postbit_onlinestatus
  • (15)postbit_wrapper
  • (1)spacer_close
  • (1)spacer_open
  • (1)tagbit_wrapper 

Phrase Groups Available (6):
  • global
  • inlinemod
  • postbit
  • posting
  • reputationlevel
  • showthread
Included Files (26):
  • ./showthread.php
  • ./global.php
  • ./includes/class_bootstrap.php
  • ./includes/init.php
  • ./includes/class_core.php
  • ./includes/config.php
  • ./includes/functions.php
  • ./includes/functions_navigation.php
  • ./includes/class_friendly_url.php
  • ./includes/class_hook.php
  • ./includes/class_bootstrap_framework.php
  • ./vb/vb.php
  • ./vb/phrase.php
  • ./includes/functions_facebook.php
  • ./includes/functions_calendar.php
  • ./includes/functions_bigthree.php
  • ./includes/class_postbit.php
  • ./includes/class_bbcode.php
  • ./includes/functions_reputation.php
  • ./includes/functions_notice.php
  • ./packages/vbattach/attach.php
  • ./vb/types.php
  • ./vb/cache.php
  • ./vb/cache/db.php
  • ./vb/cache/observer/db.php
  • ./vb/cache/observer.php 

Hooks Called (75):
  • init_startup
  • friendlyurl_resolve_class
  • init_startup_session_setup_start
  • database_pre_fetch_array
  • database_post_fetch_array
  • init_startup_session_setup_complete
  • global_bootstrap_init_start
  • global_bootstrap_init_complete
  • cache_permissions
  • fetch_postinfo_query
  • fetch_postinfo
  • fetch_threadinfo_query
  • fetch_threadinfo
  • fetch_foruminfo
  • load_show_variables
  • load_forum_show_variables
  • global_state_check
  • global_bootstrap_complete
  • global_start
  • style_fetch
  • global_setup_complete
  • showthread_start
  • showthread_getinfo
  • strip_bbcode
  • friendlyurl_clean_fragment
  • friendlyurl_geturl
  • forumjump
  • cache_templates
  • cache_templates_process
  • template_register_var
  • template_render_output
  • fetch_template_start
  • fetch_template_complete
  • parse_templates
  • fetch_musername
  • notices_check_start
  • notices_noticebit
  • process_templates_complete
  • friendlyurl_redirect_canonical
  • showthread_post_start
  • showthread_query_postids
  • showthread_query
  • bbcode_fetch_tags
  • bbcode_create
  • showthread_postbit_create
  • postbit_factory
  • postbit_display_start
  • postbit_imicons
  • bbcode_parse_start
  • bbcode_parse_complete_precache
  • bbcode_parse_complete
  • postbit_display_complete
  • memberaction_dropdown
  • pagenav_page
  • pagenav_complete
  • tag_fetchbit
  • tag_fetchbit_complete
  • forumrules
  • navbits
  • navbits_complete
  • build_navigation_data
  • build_navigation_array
  • check_navigation_permission
  • process_navigation_links_start
  • process_navigation_links_complete
  • set_navigation_menu_element
  • build_navigation_menudata
  • build_navigation_listdata
  • build_navigation_list
  • set_navigation_tab_main
  • set_navigation_tab_fallback
  • navigation_tab_complete
  • fb_like_button
  • showthread_complete
  • page_templates