www.webdeveloper.com
Results 1 to 8 of 8

Thread: use of robots.txt file

  1. #1
    Join Date
    Jun 2013
    Posts
    26

    use of robots.txt file

    Hi
    What is the use of robots.txt file in context with SEO?

  2. #2
    Join Date
    Apr 2013
    Posts
    378
    Robots.txt is used for instructing search engines..if you don't want to crawl certain pages then you can stop crawlers.

  3. #3
    Join Date
    Sep 2012
    Posts
    720
    The robots.txt file is just a simple text file as the file extension suggests. It’s created using a simple text editor like Notepad or WordPad, complicated word processors such as Microsoft Word will only corrupt the file.

  4. #4
    Join Date
    Jun 2013
    Location
    Santa Clara, CA
    Posts
    2
    Robots.txt is a text file which instructs the web crawler not to crawl, index or visit the pages that are listed in the text file. It is generally kept in the root folder of the website.

  5. #5
    Join Date
    May 2013
    Location
    chennai
    Posts
    42
    Hi Friends,

    Now i know the about robot.txt. Thanks to all. It is very useful information.

  6. #6
    Join Date
    Jan 2013
    Posts
    72
    A robots.txt file is used to restricts access to your site by search engine robots that crawl the web.
    The simplest robots.txt file uses two rules:

    User-agent: the robot the following rule applies to
    Disallow: the URL you want to block
    Basically it is used to tell the crawler which one page or file not to be crawled .

  7. #7
    Join Date
    Mar 2007
    Location
    localhost
    Posts
    2,405
    Not exactly true.

    It is purely optional on the part of the crawler to observe the robots.txt file.
    Yes, I know I'm about as subtle as being hit by a bus..(\\.\ Aug08)
    Yep... I say it like I see it, even if it is like a baseball bat in the nutz... (\\.\ Aug08)
    I want to leave this world the same way I came into it, Screaming, Incontinent & No memory!
    I laughed that hard I burst my colostomy bag... (\\.\ May03)
    Life for some is like a car accident... Mine is like a motorway pile up...

    Problems with Vista? :: Getting Cryptic wid it. :: The 'C' word! :: Whois?

  8. #8
    Join Date
    Jun 2013
    Posts
    27
    Robots.txt is a text file whichis used to guide the web crawler not to crawl, index or visit the pages that are listed in the text file. It is generally kept in the root folder of the website.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles