By now you most probably would have come across this story which has taken the internet by storm recently, especially the programming community. The story reads:  How a hosting company lost its entire business because of one line of bad code. Any person even vaguely familiar with command prompt can guess that one line:
rm -rf

(well the actual line of code as per its author was rm -rf {foo}/{bar})

The issue first came to public notice when the person responsible for this catastrophe asked for help on ServerFault (question now removed). As per the question and followed thread of comments author intended to run a script that did a few task along with deleting all files/folders inside certain folders passed as variable. Due to an error in the code, the variable got wrong value which resulted in wiping everything on the machine. Unfortunately he ran this same script on all his machines which led to deletion of everything. A complete annihilation!

Add to that he ran a web hosting company. He not only deleted his entire company code and data but also wiped clean all customer data. This affected some 1535 customers who were using his service (figures provided by him on serverfault’s thread).

Did he take backups?

Whenever a person read such stories, first thing to come across mind is – why didn’t he take backups? Well as per him, he did. He made backups on separate disks, however these disks were mounted to the main machine and hence the contagious script managed to wipe them too.

He posted a comment that read:

“All servers got deleted and the offsite backups too because the remote storage was mounted just before by the same script (that is a backup maintenance script).”

We often come across users who are trying our service and tell us at the end of trial period, while they really loved our service their hosting company provides backup and hence they may not need our service. It’s difficult to explain why you cannot blindly rely on backups done by your hosting provider but this certainly is a good example to start with.

We understand it’s a rare case scenario coupled with human error and probability of something like this happening with your premium managed hosting provider is equivalent to probability of discovering extraterrestrial life. But the important thing to notice here is there is still a probability. There are over 1 billion websites on the internet today, even mere 0.1% accounts to 1 million websites and that’s a huge number. You definitely don’t want to be one in this million group.

If something similar happens with the managed hosting provider you are signed up with, your included backups will do you no good. This hosting company just lost all its data. Yes it was because of the carelessness of the system admin but human errors can happen anywhere. There can be another similar case, where a hacker somehow breaks into your hosting company’s server and run similar script intentionally. That will affect you equally. Not only your production site is gone, also the backups.

You should never completely rely on backups by your hosts

Though there are many managed hosting companies that provide quality automated backup to their customers, one should not completely rely on these backups especially when the site in question is your main source to bread and butter. If their system is compromised, so are you and your sites. We cannot emphasise enough how important it is to have backups completely independent from your hosting servers.  

Let’s assume another case where your hosting company is hit by a major DoS attack and it went completely down for 3-4 days. Your site data may be safe but there is no way to access it. There is no certainty how soon they will recover and you cannot let your site just hang around like that. Since your backup belongs with the same hosting company, there is no way to access them either. Like it or not, you’re stuck. If only your backups were independent, you could have hosted them somewhere else meanwhile.

These are real world examples and can happen to anyone. A good backup needs to be offsite, robust, completely independent from your main servers and most importantly something you can access and deploy anywhere within minutes. We have seen enough number of times people despite having zip of their backup, running over various tech forums desperately seeking professional help to get their site restored because just unzipping it won’t bring the site back. There are various server configurations that may require fixing/updating in wake of recent disaster. Similarly a good robust backup should have an easy way to validate itself. Consider a situation where you are relying on a backup which is corrupt and you only learn this when you needed it. It’s a nightmare! While most managed hostings do provide decent backup service, these are a few scenarios where they fall flat.

Our post is not aimed to scare our readers, we just want to educate people about the importance of an independent automated backup service. One can never take their system for granted. As per the very nature of machines they are bound to crash, hacked, wiped out, melt down etc. One need to have sound backup system not just for their sake, but also for the sake of their users. And we just happen to provide one 🙂