Whew! Cerner isn’t going to live this one down easily. According to reports from the field, Cerner’s hosted EMR service was down for most of Monday, July 23, taking down both hospital and medical practice clients.
HISTalk.com, the beloved scourge of the HIT world, says the event may have taken out Cerner’s hosted network both nationally and internationally. Making things even more nasty, Cerner’s support sites seem to have been down as well.
Cerner, which used the elegantly vague phrase “unscheduled downtime” to describe the event, said that a human error was responsible for the outage but didn’t elaborate. HISTalk tweeter Richard turned said they got e-mail updates from Cerner every 15 minutes, but not every customer seems to have been attended to as closely.
Unofficially, here’s what happened. A HISTalk contributor claims that the problem was due to a mistake by a Cerner network administrator, who, when trying to update DNS records via the management console, received an error and made the change manually.
While doing so, the unfortunate administrator apparently deleted a DNS zone inadvertently. Oops! At that point, the error was replicated to all servers; and because anything using the zone couldn’t work, the tools to fix the problem weren’t available either. They then had to restore from backup which took some time.
And in the end, how bad was it? Various HISTalk.com reader reports suggested that Cerner’s service was out for all or at least a large part of the day in question. (One reader showed a graphic of a uCern ticket showing 381 minutes of outage time, or more than six hours.) Some readers reported being so frustrated that they fell back on paper processes for the day.
Now, I realize some folks are going to start tossing out questions like “What does this mean for the cloud?” I’d argue that a better question is “What does this mean for Cerner?”
Really, it’d be premature to start playing the “is the cloud to blame” game just yet. After all, we don’t all get furrowed brows and wonder aloud “what does this mean for the auto industry?” when a car crashes. This crash, like automobile accidents, may have everything to do with the specifics of Cerner’s network and little to do with the soundness of the overall technology.
While I’m a firm believer in hosted systems, it is painfully clear that not all firms are as careful as they need to be at managing these cloud installations. Changing DNS settings during the business day – not a great idea. And there are lots of other things to go wrong. Amazon has had a couple of spectacular cloud outages.
And as pointed out, you do need to have backup paper procedures.
Also, while a small doctor’s practice is far better off on the cloud, any substantial size hospital probably would do better with its own installation (and DR site).
[…] EMR and EHR has an account of what caused Cerner’s outage which affected its client sites across the USA and some […]
[…] a couple major EHR vendors had some EHR downtime, there’s been a lot of interesting discussion about how to deal with […]
[…] a couple major EHR vendors had some EHR downtime, there’s been a lot of interesting discussion about how to deal with […]