How to Block Google from Indexing your WordPress Site

So you’ve just installed WordPress on your system and are raring to go. You’re thinking of how to start and what to start with. However, before you embark on the journey of developing your website, there’s a tiny little thing you need to do – prevent Google and other search engines from crawling your site.

I know what you’re thinking. As a webmaster, one of the most important, and perhaps the most obvious thing you would want is to bring traffic to your site. And getting Google to index your site as fast as possible would surely help with that, right? Yes, it will, but you need to wait just a little longer for it. Trust me when I say that you don’t want web crawlers and robots all over your site just yet.

More often than not, you would be directly working on your live site and it is only natural for things to get messy at this stage. It is for this reason that it is advisable to temporarily block search engines from crawling and indexing your site until you’re past the initial development phase.

You might also not want Google or other search engines to get their hands on your site’s content for a variety of other reasons. So the question now is, how do you stop Google from indexing your WordPress website?

Blocking Google and Other Search Engines

Using a Robots.txt File

The most basic thing to do would be to manually create and upload a simple robots.txt file to your website’s root directory, instructing all search engines to stay away from your site and not index any part of it. The text file will carry the following syntax:

User-agent: *
Disallow: /

You can also use an inbuilt feature on your WordPress dashboard to block search engines from indexing your site. For this, you need to

1. Go to ‘Settings’, select ‘Reading’.

2. Check the box next to ‘Search Engine Visibility’ that says ‘Discourage search engines from indexing this site’. Click on ‘Save Changes’.

This automatically adds the following syntax to your site’s robots.txt file:

User-agent: *
Disallow: /

It also adds the following line to your website’s header:

<meta name='robots' content='noindex,follow' />

Although this method protects you from most of the search engine crawlers and robots out there, it isn’t a hundred percent safe.

Password Protecting your Website using cPanel

Web crawlers cannot access password-protected files. Hence, if your web host provides you with cPanel access to manage your hosting account, you can password protect your website files from your cPanel dashboard. For this, you need to

1. Log in to your cPanel account and click on ‘Password Protect Directories’;

2. Select the document root in the pop-up window and click ‘Go’;

3. Select the folder where your WordPress is installed;

4. Check the box next to ‘Password protect this directory’, type in a name for the protected directory, and click on ‘Save’;

5. Once you receive a success message, go back to create user;

6. Add a username and password, and click on ‘Add/modify authorized user’.

And you’re done! Your WordPress site is now password protected, and therefore, can’t be crawled upon by search engines.

Password Protecting your Website using a Plugin

Another way to password protect your site is by using any one of the various plugins available on WordPress itself.

All you need to do is install a plugin (it is advisable to select one that has been updated recently) and activate it. Once it’s activated, go to ‘Settings’. Enable the plugin and set your password. Click on ‘Save Changes’, and you’re done! No search engine crawler or robot can access your website, let alone index it.

Whatever your reason may be, if you want to keep search engines from crawling on your website, you can choose any of the above mentioned methods to keep your website data safe, depending on your requirements and the resources at hand.