eHealth Articles & White Papers

Prevent, Prepare and Practice: The Three Ps of IT Disaster Recovery Success

Josh Griffin - MCSE, IT Manager

From a flood to a security breach to a car accident that blows a transformer, IT disasters take many forms. The key to successfully recovering from a disaster is to develop and implement a plan. This article highlights the important steps to preparing for a disaster and shares best practices employed by Geonetric's IT department.

As you plan for a disaster, it is important to realize you are protecting your organization from the effects of the disaster - power outages, security breaches and virus outbreaks - not the disaster itself. You will never be able to foresee the disaster-that's why prediction is not one of the three P's. However, you can avert most known failures from happening.

Step 1: Preventing an outage

The first step to creating your organization's Disaster Preparedness Plan (also referred to as a Business Continuity Plan) is to prevent an outage from taking your resources offline. This requires planning - you must plan for all types of disasters, from a simple power outage to a virus outbreak. The most important step to prevention is to identify all single points of failure and security risks.

Power and cooling

At the top of the list is power and cooling. You need to have enough Uninterruptable Power Supply (UPS) to keep your critical systems running while the generators start and assume the load. Your ability to keep your equipment cool is essential to ensuring it operates efficiently.

Geonetric, as well as most hosting companies, maintains an off-site co-location center that utilizes the N+1 mythology for cooling and power. N+1 simply means the number of standard backups needed plus one. The UPS systems manage the outage while the generators start. At our facility, we run two generators, so if one fails we will not experience a power loss.

Cooling follows the same principle. You should install one more HVAC than needed to manage the load if one unit fails.

Hardware

It is important to install hardware redundancies into your hardware equipment. Hardware redundancies may include, but are not limited to, redundant power supplies, processors, hard disks and Network Interface Cards. Running a redundancy means having a ready online spare piece of equipment to replace the failed one. If your organization is running a single server, the single point of failure is the raid controller on the server. To build redundancy into this scenario, you need to identify all single points of failure and determine if redundancy is available.

Security threats

Hardware failure outages are not the only threat to your organization. External and internal security threats also have the potential to significantly harm your organization.

You should have a patch or update system that routinely performs virus updates on your computers to protect your systems. The main benefit to installing automatic patches is that you eliminate much of the human error that comes with depending on your busy staff to install updates.

Also critical to minimizing threat is a trusted intrusion detection system, such as Applied Watch. And of course, you should establish and adhere to stringent security practices.

The healthcare industry requires even more security diligence than most other industries. Most hospital and health system Web servers store patient data that must be protected. Your Internet server is open to the world so it is vital that you remain conscious of your unique threats and vulnerabilities.

One cross-site scripting attack on your website could open a hole and allow a data breach. Most automated scripting attacks exploit known vulnerabilities that have available patches and fixes.

At Geonetric, we utilize an automated update deployment process using Windows Server Update Services. We also perform weekly checks of the United States Computer Emergency Readiness Team site, which provides information on the latest reported vulnerabilities and guidance on how to protect your systems.

An intrusion detection system can identify an attack in progress and notify the IT team so they can minimize the damage. Geonetric uses a state-of-the-art intrusion detection system on our networks. This added layer or protection aides HIPAA and PCI compliance, and it detects external threats early so a bad situation does not become catastrophic.

Step 2: Prepare for the worst

It is theoretically possible to ensure every single point of failure previously identified is auto redundant and mitigate your organization's outage risk to almost zero percent. Unfortunately, most organizations do not have the budget to provide this level of redundancy, and even if they did, an unknown catastrophe could still destroy the system.

Since it is difficult to prepare for every "what if" scenario, you should try to prepare for a few possible disasters.

For example, what would happen if a car hits the transformer down the street and knocks out the power to your building while you were recycling your chillers? Your engineer likely did not configure your data center with this scenario in mind. However, if this happened, how would you react?

Or, how would you react if your intrusion detection system identified an attack in progress? What are your first, second, and third steps? How would you handle a virus outbreak?

To be prepared, you must identify possible disasters and prepare for the worst. The preparation includes a plan that outlines possible disasters and the recovery process so you can rebound as quickly as possible.

Step 3: Practice, practice, practice

Next, you need to practice your plan. Practicing your plan helps you and your staff prepare for the actual event, and it allows you to identify shortcomings in your plan and revise them before the real disaster occurs.

You have likely heard "practice makes perfect"- with enough practice your team will execute effectively when needed. At Geonetric, we re-enact outages and practice our plan on a quarterly basis.

Remember the 'Three Ps'

You can't always predict what disaster may happen, but you can control your preparation and be ready to respond. Your ability to respond effectively to the disaster can be the difference in your organization's success or loss of customer confidence.

Make sure you prevent, prepare, and practice for the next possible disaster. If a disaster occurs, you will be happy you did.