Recently, Google updated their search algorithm to favor mobile-friendly websites in mobile search and penalize those sites that do not have mobile-friendly or responsive versions. In order to help website owners confirm whether or not their site is considered mobile-friendly, Google also released a Mobile-Friendly Test Tool. The trouble is, some people are finding that their websites, which they thought were responsive or mobile-friendly, are failing Google’s test!
where these websites fail Google’s mobile-friendly test, the error message is often the same, usually something like this: “this page uses 5 resources which are blocked by robots.txt.”
What does this mean?
In order to fully understand the problem, there are two concepts that are important to grasp:
First, you should know that the purpose of a robots.txt file is to tell search engine crawlers, such as Google, what pages or files it should or shouldn’t look at on your website. A common use of robots.txt is to prevent a login or thank you page from showing up in search results. On websites that are powered by a content management system, robots.txt is often also used to block certain administrative directories on your website which are used to power your content management system.