With the meaningful use incentives, hospitals and physician practices are conducting their HIPAA Security Risk Analysis per 45 CFR 164.308(a)(1). The risk analysis focuses on the confidentiality, integrity and availability of health information. The last of these goals, availability, was brought to the forefront with Hurricane Sandy. They experienced a 2-day outage because their cloud computing vendor’s NYC data center was disabled by the flooding. As health care organizations prepare to conduct this analysis, a case study from the Hurricane Sandy is helpful.
A four-physician cardiology practice on the East Coast implemented the electronic record software from a leading physician software vendor using their cloud offering. During the risk analysis that this author conducted the vendor’s infrastructure and disaster recovery plans were reviewed. The vendor utilized 10 hosting centers geographically spread across the States. (In hindsight, 5 of these facilities are on the East Coast.) The vendor provided what appeared to be a robust disaster recovery plan. This plan included periodic replication of all data from all data centers to a backup facility, the use of the latest hardware and software offerings from major vendors, detailed protocols for declaring a disaster at a site, protocols for transferring operations to the backup site, protocols for notifying customers and protocols for transferring operations back to the original site when the disaster was over.
Hurricane Sandy demonstrated the reality that these human plans and preparations were imperfect. One of data centers was in New York City, and when the Sandy arrived, the electrical power went out. Further, the basement of the data center was flooded. Apparently the planners had not considered this scenario because the backup power generators were located in the basement, and water rendered them unusable. While the vendor publically reports that read-only backup system access was available via a different site, if this is true the practice was not aware of how to access it. After two days, the water was pumped out of the New York City site, the generators became usable, and the data center resumed operations. After this two day period the facility was able to use their system.
The net out for the small cardiology practice was that they had no system access for two business days. They were located hundreds of miles from the fury of the storm that hit the coast along New York and New Jersey. At this safe distance, they retained full electrical power, their internet connection and their phone system, and patients arrived as usual. But they experienced the dark side of cloud computing. Their vendor choked.
Fortunately, the practice had implemented a Sandy contingency plan. Prior to the storm, they had printed off medical summaries for all patients scheduled for the next few days, along with a printed copy of the patient schedule. Because of this planning and foresight, the physicians had the patient records available during the downtime. Paper progress notes were used. Because only a few days of the patient schedule were printed, the staff was unable to schedule any appointments during the downtime. When the system became available, they scrambled to telephone patients who wanted appointments. Also, the doctors had no energy to enter the encounters as structured data so they scanned the handwritten encounter note into the patient chart.
This incident is not unique. Cerner, a leading hospital EMR vendor, as reported by the LA Times and many other outlets, had an outage in July of 2012 that lasted for 5 hours and affected dozens of their hospital customers. Cerner responds that “human error” was the cause of the outage. Wired.com reported recently about outages at world-leading cloud vendors Amazon and Google, which in turn brought down other leading sites that rely on these key vendors. Cloud computing provides many benefits, but it isn’t perfect and buyers must understand the risks.
A good risk analysis will use these real world data points to quantify the risks – even major cloud computing vendors can experience disasters and/or make mistakes that lead to downtime for multiple days, or worse. To reduce the impact of downtime, healthcare organizations can invest in contingency plans so that important data is available when the primary vendor goes down. Moving beyond the risk analysis, the cardiology practice discussed here is using this experience to improve its contingency plan, which will include enhancements to their emergency-mode protocols so that they automatically maintain a local read-only backup copy of their entire database, and periodic testing of these protocols.
