Working with WordPress on a 24/7 basis the team here at Kinsta sees all sorts of different errors and warnings that clients experience. Trust us, we’ve pretty much seen it all. Whenever an error keeps popping up in our system we make sure to document it so that we can share the solution with you. What good is a premium host that doesn’t help you resolve your own issues? Today we’ll be diving into a Google Search Console error you may or may not have encountered: “The page may not render properly due to resources blocked by robots.txt.”
Don’t worry, we’ll explain what this means and how to go about fixing it on your WordPress site.
Blocked Resources
If you get a warning or error about blocked resources it typically means something is not configured correctly on your WordPress site. This is not always the case, but if you’re seeing a message about anything in Google Search Console you should always investigate it. Google provides these messages for a reason. Our recent experience with an unwarranted Google penalty definitely proves this point.
If you didn’t get a message about blocked resources you can always check your site by logging into Google Search Console. Then click on Google Index → Blocked Resources. As they state:
Googlebot needs access to many resources on your page in order to render and index the page optimally. For example, JavaScript, CSS, and image files should be available to Googlebot so that it can see pages like an average user.
These resources from this host are used by your site, but blocked to Googlebot. If Googlebot can’t access important resources on your page, the page might be indexed incorrectly. (Source: Blocked Resources Report)
Again, just because a resource is blocked doesn’t always mean it’s harming your site’s SEO. But its best practice to clean these up so when those that do impact your SEO come along you can more easily fix them without having to filter through pages of errors.
We love sharing live data with you. So in today’s example, we have an issue with multiple (400+) blocked resource errors happening (as seen below) on our own site.
When you see errors in the blocked resources section you can click on them for additional details. So we click on https://kinsta.com/wp-admin/admin-ajax.php. The admin-ajax.php file is just one example of something that may show up here. You might also see errors about JavaScript or CSS files being blocked. But fixing them generally involves the same steps.
On the page we see the following error: “The page may not render properly due to resources blocked by https://kinsta.com/robots.txt.”
Their advice given is to update the robots.txt rule to unblock the resource. If you’ve never heard of this file before, we suggest you first read our deep dive into the robots.txt file in WordPress.
You can use the Google Fetch tool to view the page as Google sees it. This can help you determine if the blocked resource is impacting the appearance of your page. Again, we simply recommend cleaning up all errors reported in here if possible.
We can see the resource it’s complaining about is https://kinsta.com/wp-admin/admin-ajax.php?action=essb_counts&...In our case AJAX is being used by our WordPress social media plugin. If we input it into Robots Testing Tool we can see that Google indeed can’t find it. This is because the /wp-admin/ directory is in fact blocked, as we’ll dive into further below.
Yoast SEO has referenced this “blocked resources” issue in a blog post containing their example robots.txt file. Basically, AJAX (admin-ajax.php) is used by some WordPress themes and plugins to add content to a page or perform a certain function. WordPress actually used to block this by default but it was fixed in WordPress 4.4 (#33156). Google can now crawl admin-ajax.php in wp-admin.
But how many of you were already running your WordPress site prior to version 4.4 coming out? Probably 99% of you. And like us, you probably have a custom robots.txt file that was tweaked by you or a developer along the way which is overwriting the new default. This means the warnings would still show up in Google Search Console unless you fix them. The answer is to simply update your robots.txt file.
Updating Your Robots.txt File
WordPress by default creates a virtual robots.txt file. However, we always recommend creating a physical one. Not sure if you already have one? Try browsing to the root of your WordPress site: https://domain.com/robots.txt. If one exists, you’ll see it. If not, you’ll get a 404 error.
We are big fans of Yoast SEO here at Kinsta, we use it on our site, and it’s the #1 SEO plugin we recommend for your WordPress site. Many don’t know you can easily create and edit your robots.txt right from your WordPress dashboard using it.
Click on SEO → Tools. Then click on “File editor.” This will not appear if you have file editing disabled. If you want to keep this disabled you can create/edit your robots.txt file via SFTP.
If you don’t have a physical file you can click on “Create robots.txt file.” This creates a physical file on your server.
If you already have a robots.txt file, it might look like this (Or it might be really long! We have seen some crazy robots.txt files):
User-agent: * Disallow: /wp-admin/
There is another line we need to add to fix the blocked resources error. So in our example, we add the following line (this is the now the default robots.txt configuration when you create a file using Yoast and WordPress on a fresh install):
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
This allows Google to now crawl it.
In regards to the admin-ajax.php file itself, you don’t have to worry about it accidentally showing up in Google or indexing because if you look at the file, it actually contains the following noindex header.
@header( 'X-Robots-Tag: noindex' );
This tag tells Google not to index it.
If you’re seeing other types of resources blocked errors, such as JavaScript or CSS, a quick way to fix them is to revert back to the standard robots.txt configuration above. The wp-content/plugins/ and /wp-includes/ are common directories we’ve seen users block by mistake which in turn sometimes causes these types of issues.
You can see below after allowing the above file in our robots.txt file our blocked resources errors resolved within a few days in Google Search Console.
Summary
Fixing errors and warnings in Google Search Console is an important part of properly maintaining your WordPress site. This helps to ensure that Google sees your site properly and indexes it. Hopefully the next time you encounter blocked resource errors, you’ll know how to better troubleshoot it!
Have any thoughts regarding blocked resources? Have you seen this on your own sites?
Brian has a huge passion for WordPress, has been using it for over a decade, and even develops a couple of premium plugins. Brian enjoys blogging, movies, and hiking. Connect with Brian on Twitter.
Thanks its really help in my site, one thing that we add a code in header can you little bit explain about it i didn’t understand
@header( ‘X-Robots-Tag: noindex’ );
Its not index my post or what in Google, What is benefit of that code only
The @header( ‘X-Robots-Tag: noindex’ ) header is automatically added to the admin-ajax.php file by WordPress, so you don’t have to worry about Google indexing it.
This is great but what if you are referring to a third party JS and they don’t allow google bots to crawl? How do you solve for the same problem?
Hello VMR, the only way to fix that would be to host the JS on your own domain instead. Besides that, there isn’t really a way to make server-level changes on a server that you don’t control.