www.webdeveloper.com
Page 1 of 2 12 LastLast
Results 1 to 15 of 20

Thread: Robot.txt File

  1. #1
    Join Date
    Feb 2014
    Location
    Dubai, UAE
    Posts
    154

    Robot.txt File

    I want to discuss about the role of robot.txt file in SEO. What is the main purpose of this file when we submit sitemap to Google? What is the difference between sitemap and robot.txt file?

  2. #2
    Join Date
    Jun 2014
    Location
    Mumbai
    Posts
    6
    If you want a search engine to include each and every URLs of your website then do not create a robot.txt file. (There is no need of robot.txt file if everything should be included in search engine.)

  3. #3
    site map is different xml sitemap and robots.txt is a text file than search engine allow information.

  4. #4
    First off, it's called robots.txt -- you don't make it plural, it doesn't do anything.

    Second, it exists to block off access to links and directories so they aren't indexed, that's all it's for. As such, it's the OPPOSITE of SEO in that it's designed to make search NOT pay attention to the content it's masking off!

    Third, it has absolutely NOTHING to do with what a sitemap is.

    Though if we're going to talk 'sitemaps' if you have every page on your site linked to by at least one other page on the site -- aka building a site properly, there is NO reason to build a sitemap or submit it to Google. It's a bunch of bekaptah nonsense that has never served a legitimate purpose.

    Finally, on the subject of BOTH robots.txt and the REL attribute, there is no such thing as "follow" and "index" no matter how many people use them out of ignorance, the only valid properties are "nofollow" and "noindex". If you want things followed or indexed, don't include the properties... or the attribute... or the robots.txt file.

  5. #5
    Join Date
    Mar 2014
    Location
    Louisville
    Posts
    4
    it exists to block off access to links and directories so they aren't indexed,

  6. #6
    Join Date
    Jan 2014
    Posts
    96
    Robots.txt is common name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private.

  7. #7
    Join Date
    May 2014
    Posts
    9
    If you don't want to crawl your website page or url then create robot.txt and add. A sitemap is a list of pages for crawlers and visitors.

  8. #8
    Join Date
    Jun 2014
    Posts
    43
    The robots.txt file is used to provide instructions about the Web site to Web robots and spiders. from accessing all or parts of a Web site that you want to keep private.

  9. #9
    Join Date
    Apr 2013
    Location
    Bangalore
    Posts
    411
    Quote Originally Posted by kiwistech View Post
    I want to discuss about the role of robot.txt file in SEO. What is the main purpose of this file when we submit sitemap to Google? What is the difference between sitemap and robot.txt file?
    Sitemap file : your website sitemap file contains all the inner pages URL of your website.So search engine will crawl all the URLs through the help of sitemap file.Whatever URL were placed in sitemap file that particular URL only will index the Search Engine.So all the inner pages of your website could be index on search engine with the help of sitemap file only.

    Robots.txt file : This totally opposite to the process of sitemap.That means If you placed the particular URL in robots file,search engine does not crawl that particular page.

  10. #10
    Join Date
    Jun 2014
    Location
    Houston
    Posts
    73
    Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention. the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.

    and
    Site map have contain the internal link of website as above discussed by the @kpkarthik.

  11. #11
    Join Date
    May 2014
    Posts
    41
    The robots.txt file prevents the pages to crawl by any robot or search engine. The XML sitemap is used for totally different purpose, it is used to tell the search engine about all your webpages.

  12. #12
    Join Date
    Jun 2014
    Posts
    2
    Quote Originally Posted by Rhodesmark View Post
    Robots.txt file consists those URLs which you don't want to be indexed. Sitemap consists URL which you want to index for search engine. So both robot.txt file and sitemap have different uses.
    I am not sure whether you can include urls in robots.txt. Usually folder paths are specified in robots.txt so that search engines does not crawl in the specified directories.

    Thanks
    Arun
    Web Design Cochin

  13. #13
    Join Date
    Feb 2014
    Location
    Dubai, UAE
    Posts
    154
    How do we write urls in robot.txt file that we donot want Google to search for?

  14. #14
    Join Date
    May 2014
    Posts
    208
    User-agent: *
    Disallow: /~joe/junk.html
    Disallow: /~joe/foo.html
    Disallow: /~joe/bar.html

  15. #15
    Robots.txt is a text file you put on your site to tell search robots which pages or folder will be crawled and with is not.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
HTML5 Development Center



Recent Articles