Breathe in, breathe out, don’t breathe...the disembodied voice wafted over the sound of ball bearings hitting metal. It was calm and confident. I, on the other hand, was not. In fact I was a little anxious…that’s what comes of being strapped to a gurney with a sensor belt around your waist and contrast dye flowing through your veins while stuffed inside an MRI machine somewhere in the bowels of the Foothills Services Building.
Breathe in, breathe out, don’t breathe…an hour is a long time to spend in an MRI machine. Your thoughts run wild. Would they remember to get me out if there was a fire? Has anyone run amok in here because they were overcome by claustrophobia (MRIophobia?) The contrast dye can cause coma or death in rare cases…am I a rare case?
What I didn’t know then but realize now is that I was a lucky case. I was one of the fortunate Calgarians who underwent a scheduled surgery or procedure before July 11, 2012. One day before to be precise.
On July 11th a transformer exploded on the top floor of the Shaw Communications building (bad) and started a fire (really bad). The Alberta Health Services (AHS) computer and backup computer were in the same building (astonishingly bad). The explosion and fire knocked out the AHS computer and set off the sprinkler system which knocked out the backup computer (like we couldn’t see that one coming). The AHS IT system went down for 36 hours. It’s still not completely functional.
So what’s the impact of this outage? According to Health Minister Fred Horne, it’s no biggie. “What’s important in this case is that no urgent or emergent healthcare services were affected”.*
Think about that for a moment…how does Mr Horne know this to be true?
Mr Horne would have no knowledge of who required urgent or emergent healthcare services and who didn’t unless he reviewed the patient records. These records are housed in the AHS computer. The AHS computer and the AHS backup computerwere rendered useless as a result of the fire and water damage caused by the explosion at the Shaw Communications building. So it would be safe to assume that Mr Horne had no evidence to support his statement other than this: the urgent or emergent cases that came in through emergency were not “affected” (whatever that means) by the outage.
Contrast Mr Horne’s complacent reaction with that of Dr Fernandes. AHS called Dr Fernandes’ office to advise him of urgent medical results for two of his patients. Dr Fernandes returned the AHS call an hour later, asking for the patients’ names, but by then the AHS computers had gone down, taking the patients’ identities with them.
Presumably Minister Horne could argue that advice of urgent medical results is not the same as being in need of “urgent health-care services”,however Dr Fernandes was not so cavalier. His practice includes elderly patients, organ transplant recipients and patients on dialysis. Rather than risk the health of two of his patients his office spent hours phoning hundreds of patients to check on their welfare.
Minister Horne’s refusal to accept responsibility for this disaster is shocking. Any enterprise with an inkling of common sense has a business continuity plan to protect itself from IT system failure. And yes, I know that AHS outsourced its IT computer services to IBM (to the tune of $100 million/year I might add)**but this does not absolve AHS from ensuring that the IBM service contract obligates IBM to provide disaster recovery services and real-time backup services. To do anything less for an IT network that is critical to the health and welfare of Albertans is not just poor business practice, it’s irresponsible.
Mr Horne’s position appears to be: it’s OK, nobody died. His facile response is typical of the PC government’s refusal to take accountability for its actions, but it really doesn’t cut it for the 110 patients who’ve had their elective surgeries cancelled and the 298 patients who’ve had their MRIs, CAT scans and other tests cancelled. All of them are back in the wait list queue.
Furthermore, Mr Horne’s response is woefully inadequate for any Albertan who expects an ounce of compassion from the government minister charged with providing adequate healthcare to Albertans.
* Calgary Herald, July 14, 2012
**Calgary Herald June 14, 2012, A9
Susan on the Soapbox 
You’re not the only one shocked at the failure to have adequate backup and recovery for the critical IT services.
Susan, you gave a pretty vivid description of what the MRI experience was like. Fortunately, it sounds like the computer failure didn’t affect you. Still, it’s another one of those things where people think it’s crazy to ask questions like, “What if there’s a fire alarm, what if there’s an equipment failure, what if there’s a fire in the building that houses the computers and the backup systems are in there too?”.
Thanks for the information and analysis.
As you pointed out in a recent Goldfish post http://goldfishinasharktank.com/ the questions people should be asking are the ones they think it’s crazy to ask. I wonder whether one of the 90,000 people who work for AHS did ask these questions but was told it’s too expensive to put in a backup system off site somewhere. Makes absolutely no sense to me, but hey, Mr Horne promised a government-wide review of the outage. Knowing how averse the PCs are to taking responsibility for stupid decisions, I don’t hold out much hope that this review will be very illuminating.
PS I haven’t heard anything from anyone about whether my MRI results were caught up in the outage, hopefully not otherwise it’s another 9 months on the waiting list!
Well, my personal opinion is they cheaped out. I work for a company that runs their computer system on 6 servers. This, basically, in the computer world would be called old school as a company our size would be better served with an updated system run by one large server. So, why 6 servers? The system started many years ago with one server and as the company grew more servers were required.
The division of the computers programs was decided on what “needed” to be running 24/7; so how do you best accomplish this? The computer technologist grouped the systems, including a generator that would keep certain computers always running, and the others could be shut down at times for maintenance. He incorporated 2 online providers so if one went down the other was still operating. The final addition was an external off-site backup that, once a week, backed up the data so should the building burn down you were only ever one week short of a total reconstruction.
So, why does our system need to operation 24/7 because we are associated with the RCMP and they require it. If the RCMP can dictate the need for 24/7 service why can’t health care. What happens if an organ donor suddenly dies and you can not access the health care system to find out where his organs should go. As the organs have a limited transplant time, it would be awful to know that your loved one died because a computer did not have suffient back up or went down.
I work for an independent owner operated company so all these additions come out of their profits. Government tax dollars (my tax dollars) should be spent on better health care services, whether it be doctors, nurses or even computers.
These are all excellent points Rose Marie, which illustrate how even a small company with an ounce of foresight can ensure that it will be fully operational 24/7. It is difficult to fathom the thinking that allowed AHS to get itself into this mess. At first the public thought Shaw Communications was the culprit, but we quickly realized that Shaw is just a service provider, it was up to AHS to determine its needs (24/7 availability and absolutely NO downtime) and do exactly what your company to ensure that there were never any service interruptions. The fact that AHS failed to do this simply boggles the mind.
May I give AHS your name if they ask for assistance…just kidding! 🙂
Great comments and great questions.
I am not surprised with any of it. As always there will be a review and then all the big wigs sit down and talk about something they know nothing about and then everything goes back to the same until another fire hits and the same situation repeats. The fact is that there are always solutions to these issues but they are not cheap and the final decision is always the cheap one. I personally just wished these people would just admit this and just take the consequences. If we cannot afford to have real backup systems then just admit it and take what comes with that decision. Now what infuriates me is this atittude that they expect to have the top service when they do not pay for it.
Fred Horne does not have a clue of what he is talking about and neither does most of the people around him. Believe me I know this to be the case.
Just like the doctor inquiry and all the secrecy around everything, another investigation will be done and we will not know about anything because it is confidential.
When something like this happens I have a laugh thinking about when people mention that we are ready even for a nuclear war.
I had a hard time believing that you were surprised that AHS failed to deal with what can be considered a relatively small problem event. YEP that is the reality. I am quite certain it is not much different with many other services we think are definitely ready for any possible disruption. Do not bet on it. Furthermore this is not just a Government domain situation. Look at the Enbridge pipeline. They knew very well that the pipeline had cracks. What did they do? NOTHING. It is cheaper to damage the environment. Then they call those who are very concerned with possible spills all along the new pipeline they want to build to the coast of BC, extremists, and even terrorists, anti-development … etc.
By the way did you see Alison Redford in Calgary or is she still in China? For a while I thought she had become the Energy Consultant for the president of China. It seems all the MLAs that did not get elected became energy consultants. That probably explains why we need deregulation to give all these people good paying jobs. After all they have done such a great job that Alberta is close to be the highest energy priced oild rich area in the world. We are certainly the most expensive in Canada. The amazing is that despite all this evidence of horrible management of our resources by the previous governments of Ralph Klein, Ed Stelmach and now Alison Redford who seems to not do anything any different, the almost obliteration of the Heritage Trust Fund, ruining the Health Care System…..;.etc – they are considering the Order of Canada for Ralph Klein.
You are right Susan this is all mind boggling and we keep voting them in. I can only imagine what Peter Lougheed thinks about all of this. 🙂
There was another small fire at the Shaw Communications building today–once again demonstrating how critically important it is for any company, large or small, to have a backup computer system somewhere off-site. This is such a basic concept that no one can believe that the AHS made such a stupid mistake. I agree with the explanation provided by you and other readers–the government was trying to save money. Now they’re trying to pin the blame on Shaw, but let’s face it, all the government had to do was what every other company does and pay for a server and a back up server which would be located somewhere other than the Shaw building. It will be interesting to see whether the opposition parties and the public lets the PCs get away with it.
Susan there is nothing more powerful than a personal experience. Thankyou for helping us make the link between an ‘unfortunate oversight’ and a ‘near-death experience’. Saccharine reassurrances from the Minister of Health are standard fare now thanks to the $14M Public Affairs Bureau that sanitizes and minimizes all bad news and dismisses the critics. Every minute of every day doctors and patients are making life-changing decisions based on the information at hand. I have no doubt that many Albertans’ health were adversely affected due to another area of negligence – namely ensuring a basic, safe alternative energy sources for critical IT. As a medical colleague quipped after the PC leadership decision in favor of Allison Redford – “You can change the broom but we still have the same rug under which to sweep the failures”.
David, the PAB will be working overtime on this one, the government is already trying to point the finger of blame at Shaw, but ultimately it was the government’s decision, not Shaw’s or IBM’s, to go with the arrangement that resulted in this catastrophic failure. The only way that we can ensure the PCs don’t get away with it is to publicize the stories of those people who were caught by the outage and didn’t get the treatment they needed or got treatment that was less than optimal. What a pathetic incident, made all the more galling by the way Mr Horne brushed it off. It’s like your friend said: just sweep it under the rug.
Here is a good video to watch on LIBOR and the rest of the robbery.
http://www.truthdig.com/avbooth/item/bill_moyers_talks_greedy_banks_20120716/