Shaw Court on fire

[Resolute 14]

Originally Posted by mykalberta

Foothills hospital doesnt have its own local server?

WAN disconnect should not disrupt LAN connectivity, thats just poor network planning.

Uhh, the world's best network would still fail to access an enterprise system that is offline. Nehkara's problems aren't related to their network plan. The problem is that the central servers hosting his software are gone at the moment.

The failure here (and in my company's case) is a lack of redundancy hosted at another site. But we can survive without SAP for a couple days. I'm not sure how critical Nehkara's systems are, but if they were of the type that would impact client care, I suspect AHS would have had that redundancy in those systems.

[Rathji]

Internet is up at work now, so I imagine most downtown businesses are connected as well.

[Burninator]

Originally Posted by Regulator75

Dumb Dumb Dumb...

All the data centers / critical areas that I've worked in, all had Halon or some sort of fire suppression system.

Water + data = major failures.

Halon isn't very economical for even a moderately sized data center as it's insanely expensive and you would need a lot of it. A water sprinkler system is a fire suppression system so I am not sure what you are referring too.

[Rathji]

Plus automatic Halon has the problem where it instantly kills everyone in the room as well, which can be problematic.

[Zee]

Originally Posted by Rathji

Plus automatic Halon has the problem where it instantly kills everyone in the room as well, which can be problematic.

Halon has been replaced by Inergen (in most places) or water in most places because of this...

Overall, there appears to be a complete lack of business continuity and disaster recovery planning for some very critical services.

[Locke]

Originally Posted by Kavy

X929 has been playing off and on Classic Rock and County Music in honor of County105 and Q107 which went down with the explosion, despite being owned by a rival company.

I think its pretty funny when I hear Pink Floyd's Comfortably Numb followed by an voice over going "X929, Calgary's New Rock Alternative"

I thought this was hilarious. I dont mind the odd song, but they were getting their digs in too.

[worth]

Originally Posted by Burninator

Halon isn't very economical for even a moderately sized data center as it's insanely expensive and you would need a lot of it. A water sprinkler system is a fire suppression system so I am not sure what you are referring too.

If it's good enough for Cyberdyne Systems Corporation, it's good enough for Shaw.

[Slava]

In terms of pure risk management this looks incredibly pathetic. First, to not anticipate the problems with water and their equipment is basically amateur hour. That's a pretty basic screw-up there! Next you have to wonder why this is a choke-point in the system? Don't they have any duplication whatsoever? I don't know about how this stuff works (can clearly there are people who are more qualified to comment on that here), but its a basic, basic premise of risk management. Sort of like having executives fly on separate planes, or having an offsite back-up. We're not talking about some amazingly high level concept here.

[Madman]

Overnight work looks like it has repaired the problem and they are awaiting Fire approval.

Shaw expects all services to be restored by end of day following major fire

Read more: http://www.calgaryherald.com/Shaw+ex...#ixzz20QK9pEhN

[Ace]

Our business phones in NE calgary are still down...

[Bobblehead]

The problem with multiple site redundancy is it gets very complex and that means expensive.

Especially with things like health care and municipal services - how much more are people willing to pay in order to be able to withstand a rare disaster? And when there may be little to no other observable benefit it can be a tough sell - "You want to spend hundreds of thousands of dollars a year on the off chance something might happen?"
Outside of 911, it sounds like most of the other things affected are non-critical. A PITA that they aren't available, but still nothing that people can't live without for a couple days.

As for sprinkler systems - sure it would be great if every building planned for a better solution. But look how fast tech has taken over and how much power you can get now; a lot of buildings were built without any concept of data centres like this and even retro-fitting is a costly, might never need it, endeavour. Then you get the whole, it just started with one computer so we didn't worry about it and without noticing it grew into the mass it is today.

If the data centre in question is planned to be relocated (is that the construction off Barlow/32nd beside their other huge building?) then they got really unlucky - it had lasted this long - or they pushed their luck a bit too far.

[Rathji]

Originally Posted by Slava

In terms of pure risk management this looks incredibly pathetic. First, to not anticipate the problems with water and their equipment is basically amateur hour. That's a pretty basic screw-up there! Next you have to wonder why this is a choke-point in the system? Don't they have any duplication whatsoever? I don't know about how this stuff works (can clearly there are people who are more qualified to comment on that here), but its a basic, basic premise of risk management. Sort of like having executives fly on separate planes, or having an offsite back-up. We're not talking about some amazingly high level concept here.

Obviously they do, to a large degree, or many businesses would not have connections. The fact that it takes real time to do disaster recovery is a fact of life, unless you honestly expect them to fully duplicate everything just in case something went wrong.

That's like for your plan, you actually have 2 offices across the street from each other, just in case yours burns down. Of course you wouldn't be doing that, but in reality, you might have a data backup offsite, a personal computer you could use and you might need to see clients in a hotel room. This is a much more reasonable expectation, imho, but it would take time.

That said, I have no idea what total function the area impacted by this fire served so I really can't comment, other than to say I am surprised that there are still businesses without internet today as a result of this, since the one thing I would fully expect to be redundant is the network routing gear.

[mykalberta]

Originally Posted by Resolute 14

Uhh, the world's best network would still fail to access an enterprise system that is offline. Nehkara's problems aren't related to their network plan. The problem is that the central servers hosting his software are gone at the moment.

The failure here (and in my company's case) is a lack of redundancy hosted at another site. But we can survive without SAP for a couple days. I'm not sure how critical Nehkara's systems are, but if they were of the type that would impact client care, I suspect AHS would have had that redundancy in those systems.

Here is what I was referring to specifically

Originally Posted by Nehkara

Things are just a mess here at work (Foothills Hospital)....
As a backup to that we do things on tape and save them to a network drive.

Network drive is gone.
...

To not have local server access at a facility the size of a major hospital (probably second largest in the province) as a backup is poor network design on the part of AHS imo.

I am sure the reasoning is cost related (Big Blue as a contractor who leases 3 floors at Shaw) because it would require multiple onsite support persons at Foothills and you dont get the scalability and downward pressure on costs that centralizing everything can provide.

AHS designed a network that has a single point of failure and no local backup plan. Dont get me wrong, alot of companies work like that; if MacDonalds or Dairy Queen loses their telco access then their debit machines wont work. I just assumed AHS would have planned their network some some sort of local file server backup.

[Resolute 14]

It would depend on what network drive he was referring to, what it hosts and how critical it is. That he's lost his drive does not mean AHS has lost all network shares, or even critical shares.

[cam_wmh]

Originally Posted by Slava

In terms of pure risk management this looks incredibly pathetic. First, to not anticipate the problems with water and their equipment is basically amateur hour. That's a pretty basic screw-up there! Next you have to wonder why this is a choke-point in the system? Don't they have any duplication whatsoever? I don't know about how this stuff works (can clearly there are people who are more qualified to comment on that here), but its a basic, basic premise of risk management. Sort of like having executives fly on separate planes, or having an offsite back-up. We're not talking about some amazingly high level concept here.

[Slava]

Originally Posted by cam_wmh

OK, so what's the solution? I told you my limitation right off the hop, but you just take that and reinforce it. I might be in that small minority that thinks that making sure that services like 911 and AHS should be accessible in an emergency, regardless of whether I can tell you how that should be made possible.

[ken0042]

Originally Posted by Slava

I might be in that small minority that thinks that making sure that services like 911 and AHS should be accessible in an emergency, regardless of whether I can tell you how that should be made possible.

Part of that is the media sensationalizing things a bit. 911 was available from any network other than Shaw. So Telus landline, cell phone, etc. The impact was about 30,000 users out of ~400,000 households. And I would say that most of the Shaw Phone customers have a cell phone as well. However the media was calling it "911 unavailable." Then what compounded the issue was everybody with a Telus phone was calling 911 just to see if they could get through.

As for AHS services, this meant that if you went to the hospital they would make you fill out the entire form like they did 10 years ago; instead of asking for your Health Care Number and asking if your address had changed.

[Handsome B. Wonderful]

Originally Posted by Rathji

Plus automatic Halon has the problem where it instantly kills everyone in the room as well, which can be problematic.

Halon does what now?

[Slava]

Originally Posted by ken0042

Part of that is the media sensationalizing things a bit. 911 was available from any network other than Shaw. So Telus landline, cell phone, etc. The impact was about 30,000 users out of ~400,000 households. And I would say that most of the Shaw Phone customers have a cell phone as well. However the media was calling it "911 unavailable." Then what compounded the issue was everybody with a Telus phone was calling 911 just to see if they could get through.

As for AHS services, this meant that if you went to the hospital they would make you fill out the entire form like they did 10 years ago; instead of asking for your Health Care Number and asking if your address had changed.

Well the sensationalism is definitely a factor and I probably fell victim to that to an extent. When I heard on the radio that they advised people to have their neighbour call 911 for them if they needed it though, I had to laugh.

[bizaro86]

Originally Posted by ken0042

Then what compounded the issue was everybody with a Telus phone was calling 911 just to see if they could get through.

Did people actually do that? What a bunch of morons. You don't call 911 as an experiment to see if they're there, it's for legitimate emergencies.

edited to add: I'm pretty sure my shaw phone line came with some sort of disclaimer about 911 during power outages and whatnot.