use of robots.txt file
What is the use of robots.txt file in context with SEO?
Robots.txt is used for instructing search engines..if you don't want to crawl certain pages then you can stop crawlers.
The robots.txt file is just a simple text file as the file extension suggests. It’s created using a simple text editor like Notepad or WordPad, complicated word processors such as Microsoft Word will only corrupt the file.
Robots.txt is a text file which instructs the web crawler not to crawl, index or visit the pages that are listed in the text file. It is generally kept in the root folder of the website.
Now i know the about robot.txt. Thanks to all. It is very useful information.
A robots.txt file is used to restricts access to your site by search engine robots that crawl the web.
The simplest robots.txt file uses two rules:
User-agent: the robot the following rule applies to
Disallow: the URL you want to block
Basically it is used to tell the crawler which one page or file not to be crawled .
Not exactly true.
It is purely optional on the part of the crawler to observe the robots.txt file.
Yes, I know I'm about as subtle as being hit by a bus..(\\.\ Aug08)
Yep... I say it like I see it, even if it is like a baseball bat in the nutz... (\\.\ Aug08)
I want to leave this world the same way I came into it, Screaming, Incontinent & No memory!
I laughed that hard I burst my colostomy bag... (\\.\ May03)
Life for some is like a car accident... Mine is like a motorway pile up...
Problems with Vista? :: Getting Cryptic wid it. :: The 'C' word! :: Whois?
Robots.txt is a text file whichis used to guide the web crawler not to crawl, index or visit the pages that are listed in the text file. It is generally kept in the root folder of the website.
Users Browsing this Thread
There are currently 1 users browsing this thread. (0 members and 1 guests)