dcsimg
www.webdeveloper.com
Results 1 to 4 of 4

Thread: Robot.txt and its usage

  1. #1
    Join Date
    Jun 2017
    Location
    Lucena City, Quezon, Philippines
    Posts
    7

    Robot.txt and its usage

    What is a robot.txt and how can it help me on building SEO strategies?

  2. #2
    Robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website. The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users. The REP also includes directives like meta robots, as well as page-, subdirectory-, or site-wide instructions for how search engines should treat links (such as “follow” or “nofollow”)

    Source:MOZ

  3. #3
    Join Date
    Mar 2012
    Posts
    3,786
    Robots.txt serves little or no SEO purpose. It merely excludes those parts of the site (if any) that you want the search bots to ignore. However, given that well-behaved bots will only crawl pages that are directly or indirectly linked to the home page, why would you link such pages anyway? Whereas, badly behaved bots are unlikely to obey the robots.txt exclusions. Indeed it may assist them by identifying private areas on the site!
    Last edited by jedaisoul; 06-21-2017 at 01:27 PM.

  4. #4
    Join Date
    May 2015
    Posts
    76
    Robots.txt serves a little purpose where some one can mention what pages are to be crawled by the crawler and what not. Suppose you have a e-commerce website and did not want to share information about the checkout page of the website where the sales and purchase happened at that place robots.txt is helpful.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles