dcsimg
www.webdeveloper.com
Page 2 of 2 FirstFirst 12
Results 16 to 20 of 20

Thread: Robot.txt File

  1. #16
    Join Date
    Jul 2014
    Posts
    26
    If your website has a robot.txt file, then it helps you a lot to get ranked on search engine. A robot.txt is a simple text file which tells search engine crawlers about your website and which page have to crawled and which have to be ignored. Sitemap and robot.txt both plays an important role, both serve differing but in complementary purposes.

  2. #17
    Join Date
    Mar 2007
    Location
    localhost
    Posts
    2,506
    robots.txt file is a reference for web crawlers who read the file and follow its instructions when crawling a site and it is completely optional.

    If you have directories (folders) on the server you don't want crawled then you will have to use server side .htaccess files which the server reads when a request is made on that folder. Namely you are looking for the settings to not allow contents listing / directory listing if you use .htaccess
    Yes, I know I'm about as subtle as being hit by a bus..(\\.\ Aug08)
    Yep... I say it like I see it, even if it is like a baseball bat in the nutz... (\\.\ Aug08)
    I want to leave this world the same way I came into it, Screaming, Incontinent & No memory!
    I laughed that hard I burst my colostomy bag... (\\.\ May03)
    Life for some is like a car accident... Mine is like a motorway pile up...

    Problems with Vista? :: Getting Cryptic wid it. :: The 'C' word! :: Whois?

  3. #18
    Join Date
    Mar 2007
    Location
    localhost
    Posts
    2,506
    When I say completely optional, I mean that it is up to the web crawler to stick to your request, most will not and they will crawl the entire site.
    Yes, I know I'm about as subtle as being hit by a bus..(\\.\ Aug08)
    Yep... I say it like I see it, even if it is like a baseball bat in the nutz... (\\.\ Aug08)
    I want to leave this world the same way I came into it, Screaming, Incontinent & No memory!
    I laughed that hard I burst my colostomy bag... (\\.\ May03)
    Life for some is like a car accident... Mine is like a motorway pile up...

    Problems with Vista? :: Getting Cryptic wid it. :: The 'C' word! :: Whois?

  4. #19
    Join Date
    Jun 2014
    Location
    Canada
    Posts
    3
    The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index.

  5. #20
    Join Date
    Jul 2014
    Posts
    63
    Robot.txt gives idea to search engines about which content of the website should be crawled and which have to be ignored. It is a text file and it is placed on every websites root directory. Sitemap and robot.txt both are important for any website.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles