Though it’s not something that many people think about, robots.txt files made it easy for businesses to control which of their pages were indexed by Google. At times, these NoFollow rules were used to hide pages that were under construction, old pages that were archived, or pages that were meant for internal audiences.
Without updating their site’s robots.txt file, there’s a chance that unwanted pages may appear in the search results. Many platforms, like WordPress, used robots.txt files to prevent pages from being indexed. This means that websites may be unaware that their sites are affected by the new policy at Google.
Website owners who use WordPress won’t have to worry about the wrong pages being indexed once WordPress launches its 5.3 update. This update will switch from using robots.txt file to meta tags. This change will bring WordPress users in line with Google’s best practices for No Follow links.
In a post about the upcoming change, Peter Wilson wrote, “In WordPress 5.3 the method used to discourage indexing will change on sites enabling the option “discourage search engines from indexing this site” in the WordPress dashboard. These changes were made as part of ticket #43590. These changes are intended to better discourage search engines from listing a site rather than only preventing them from crawling the site.”
Ensuring that your website has the proper setup to prevent unwanted indexing is important for the SEO and digital strategy for a website. There are legitimate reasons to hide a page from search engines, so website owners need to check to see which method their platform uses.
The WordPress update is a reminder that it’s vital for business owners to keep their site updated with the latest releases from WordPress. These updates fix critical flaws and bring the platform up-to-date with the latest trends on the internet. If it’s been a while since someone has updated your WordPress site, notw is a good time to have it checked.
The WordPress 5.3 update is expected to be released in November 2019. In the meantime, website owners should double check to make sure no critical information is getting scanned by crawling search engines. Using robots.txt files for No Follow links was the standard for many years.
Though it may seem strange to change a system that was working, Google has its reasons for ending support for NoFollow directives on Robots.txt files. As one SEO journalist explained, “WordPress relied on Robots.txt for blocking the indexing of a website because that’s how everybody kept pages from showing in Google’s search results. That was the standard way of doing it. Yet even though everybody did it that way, as has been explained, it was an unreliable approach.”
Make sure your website is using the robots meta tag to hide content from Googlebots. And make sure you update your site to WordPress 5.3 when it becomes available. For more information about changes to Google, read this article on Google’s crackdown on inauthentic click activity.