www.webdeveloper.com
Results 1 to 4 of 4

Thread: Major role of Robots.txt file in a website

  1. #1
    Join Date
    Mar 2013
    Posts
    51

    Major role of Robots.txt file in a website

    Robots. text file is considered as a useful convention to prevent cooperating web robots and web crawlers from accessing all or part of a website or its content for which we donít want to be crawled and indexed but publicly viewable. It is also employed by search engines to archive and categorize website and to generate a rule of no follow regarding some particular areas of our websites.. pls share your experience..

  2. #2
    Join Date
    Apr 2013
    Location
    Bangalore
    Posts
    408
    If we adding the particular and major URL in robots.txt file,Google does not crawl that particular URL of our website.That is the main role of robots.txt file.

  3. #3
    Join Date
    Sep 2013
    Location
    Milpitas, CA
    Posts
    58
    In robots.txt file we put those links which we want should not be crawled and indexed by the Google but we want to be viewable by the users.

  4. #4
    Join Date
    Nov 2013
    Location
    Chittagong
    Posts
    1
    If you want to create a website with responsive design and seo services then you can get many website design company on the net. But you have to ensure before creating your site to provide them this responsibilities. They help you to create robots.txt file and indexing your site on google.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles