dcsimg
www.webdeveloper.com
Results 1 to 12 of 12

Thread: Robot.txt and its usage

  1. #1
    Join Date
    Jun 2017
    Location
    Lucena City, Quezon, Philippines
    Posts
    7

    Robot.txt and its usage

    What is a robot.txt and how can it help me on building SEO strategies?
    Share on Google+

  2. #2
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”)

    Source:MOZ
    Share on Google+

  3. #3
    Join Date
    Mar 2012
    Posts
    3,848
    Robots.txt serves little or no SEO purpose. It merely excludes those parts of the site (if any) that you want the search bots to ignore. However, given that well-behaved bots will only crawl pages that are directly or indirectly linked to the home page, why would you link such pages anyway? Whereas, badly behaved bots are unlikely to obey the robots.txt exclusions. Indeed it may assist them by identifying private areas on the site!
    Last edited by jedaisoul; 06-21-2017 at 01:27 PM.
    Share on Google+

  4. #4
    Join Date
    May 2015
    Posts
    80
    Robots.txt serves a little purpose where some one can mention what pages are to be crawled by the crawler and what not. Suppose you have a e-commerce website and did not want to share information about the checkout page of the website where the sales and purchase happened at that place robots.txt is helpful.
    Share on Google+

  5. #5
    Join Date
    Sep 2016
    Location
    Indore, India
    Posts
    9
    Robots.txt is a text file, and it is used to exclude/block sensitive info, web pages or directories of any website from searching.
    If you don't want to show or index a web page/directory during searching on Google or other search engines, then you can mention it on robots.txt file.
    Share on Google+

  6. #6
    Join Date
    May 2016
    Posts
    130
    Hi,

    I have doubt regarding robots file creation. If we exclude or include the CSS file and Image folder in robots.txt file.
    Share on Google+

  7. #7
    Join Date
    Jun 2017
    Posts
    16
    Robots.txt is a file where we give instructions to crawler which page to crawl and which not.
    e.g. pages like **Links removed by Site Staff so it doesn't look like you're spamming us. Please don't post them again.** , disclaimer and privacy policy.
    We can stop crawler for crawling our website by giving instruction in Robots.txt file.
    Last edited by Train; 06-24-2017 at 07:29 AM.
    Share on Google+

  8. #8
    Join Date
    Apr 2017
    Posts
    25
    Robots.txt is text files mainly report to crawler that it is permitted any website directory. The use of robots.txt is to disallow crawlers from visit private folder or content that gives them no extra information.
    Share on Google+

  9. #9
    Join Date
    Jun 2017
    Posts
    53
    Well you can just look it up on wiki, there is like 5 posts and each give you different answer that kind of suck.
    Share on Google+

  10. #10
    Join Date
    Apr 2017
    Location
    Bangalore
    Posts
    51
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.
    Share on Google+

  11. #11
    Join Date
    Jun 2017
    Posts
    53
    Hey just read it http://www.robotstxt.org/robotstxt.html it's way better explained then anyone have said here!
    Share on Google+

  12. #12
    Join Date
    Oct 2013
    Location
    Sheboygan, Wisconsin
    Posts
    1,427
    “Closing due to large number of replies”
    Share on Google+

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles