Shaw Court on fire

[Hockeyguy15]

Originally Posted by Notorious Honey Badger

I don't hate Shaw. What gave you that idea? If TELUS did this I'd be just as unimpressed with peoples surgeries being cancelled because they lacked proper redundancies

It's easy complain and jump to conclusions when we don't have all the facts. I have a friend who is a nurse and she has said nothing about surgeries being cancelled, only that she hopes the system comes back on soon so there is less paperwork.

Your post just seems like you want to pile it on Shaw. Saying they needed redundancies, when they did have them and it was a catastropic event. Then you complain about their explaination saying it wasn't good enough.

And really, do you want them analyzing to death what happened or do you want them to fix the problem first then analyze?

[mykalberta]

Originally Posted by Slava

OK, so what's the solution? I told you my limitation right off the hop, but you just take that and reinforce it. I might be in that small minority that thinks that making sure that services like 911 and AHS should be accessible in an emergency, regardless of whether I can tell you how that should be made possible.

It depends what the problem is you are trying to fix and how AHS is designed.

To fix a single point of failure is to determine if a backup is cost effective and just exactly what level of service is you can live with at minimum for a certain amount of time.

Example, if AHS uses a shaw connection to their contractors (IBM, which sounds is the major culperate here, not AHS directly but IBM acting as a contractor for AHS) then you install a second Telus/Enmax Envision line and you use both lines in //, example foothills hospital has a 10mbps line from shaw and a 10mbps line from Enmax giving them 20mbps at peak and 10mbps at times of issue and the network engineers can setup traffic filtering to limit bandwidth to only essential traffic (but this should all be in an emergency response plan).

If you want to fix IBM going down as your single point of failure, that requires likely a data routing redundancy that if AHS is one unit is likely cost effective by having a data centre in Calgary and Edmonton, if its just AHS acting for Calgary health region its likely not as cost effective. Again it all depends how AHS is designed, if Foothills had to cancel surgeries etc that seems fairly major problem either via network planning or emergency planning (are there no paper copies of work orders, is everything computerized).

911 Shaw telco customers, cant really fix that, phone numbers always will have a single point of failure - the phone itself. If 911 is extremely important for you so much so you are willing to pay more, Telus basic phone will always be a better option for 911 service than Shaw phone will be - you can use Telus when the power goes out but you cant use Shaw.

I personally think Shaw will be losing at least 1/3 of their dual service corp clients biz for those especially who have data and voip through them. Its not Shaw's fault per say but its simply prudent planning to have one company run one system and another run one so if you lose phone (voip) you at least have internet (email). The suits who have 2 letter designations for titles normally dont care about that until a pie lands smack in their face. The only good thing is it happened durring stampede so actual business disruption to the decision makers was likely limited.

I know of at least 3 mid companies(500-1k users) who where completely dead on everything but internal calling.

[Notorious Honey Badger]

Maybe your friend who's a nurse didn't have any examples. But my Mom isn't lying when she said surgeries are being cancelled and have to rebooked. I mean I guess that's a meaningless source to you but I'm not here to convince you.

I'm not piling on either, you're just being defensive. What happened here shouldn't have happened and should have been covered by a better system of redundant backups. This isn't a liberal attacking a PC for bad policies and making it political, this is serious stuff. People didn't have 9-1-1. Hospitals were affected. It's really too bad you're trying to spin my posts as "hate".

[Rathji]

So I am confused, does Shaw run hosted services for AHS, or it is just the connection to <inserr other service name here> that has been disrupted?

Seems to be claims of a bit of both here.

[ken0042]

Originally Posted by darockwilder

It's a lot of extra work for an already stressed system.

I don't disagree with you on that. I have worked places where we depended on computers, and when they went down the workload increased. I can imagine there is an impact here.

However the spin by the media was almost that hospitals had shut down. Wording like "911 and 311 are both down, Alberta Heath care systems are all down and the city has enacted the The Municipal Emergency Plan." While the situation was far from ideal; it wasn't as much a doom and gloom situation that they made it out to be.

Look at what Slava thought. He's a smart guy; just not a geek like some of the rest of us. He thought 911 and AHS were both down last night.

[darockwilder]

Originally Posted by ken0042

I don't disagree with you on that. I have worked places where we depended on computers, and when they went down the workload increased. I can imagine there is an impact here.

However the spin by the media was almost that hospitals had shut down. Wording like "911 and 311 are both down, Alberta Heath care systems are all down and the city has enacted the The Municipal Emergency Plan." While the situation was far from ideal; it wasn't as much a doom and gloom situation that they made it out to be.

Look at what Slava thought. He's a smart guy; just not a geek like some of the rest of us. He thought 911 and AHS were both down last night.

Yeah, the spin from media was "disaster" and the spin from AHS was "business as usual". Somewhere in the middle lay the truth.

Hopefully it will be fixed soon.

[Hockeyguy15]

Originally Posted by Notorious Honey Badger

Maybe your friend who's a nurse didn't have any examples. But my Mom isn't lying when she said surgeries are being cancelled and have to rebooked. I mean I guess that's a meaningless source to you but I'm not here to convince you.

I'm not piling on either, you're just being defensive. What happened here shouldn't have happened and should have been covered by a better system of redundant backups. This isn't a liberal attacking a PC for bad policies and making it political, this is serious stuff. People didn't have 9-1-1. Hospitals were affected. It's really too bad you're trying to spin my posts as "hate".

I never said your mom was lying, I just said that my source hasn't mentioned anything like that. What were the surgeries that were rebooked? I think rebooking not serious surgeries is a good thing. Right now you probably have a bunch of man power being lost because doing things takes more time. Doctors and nurses should be focused on giving care to the people who need it most and I would bet that people who really need the care are getting it. If that means rebooking a knee surgery here and there a few days later than I'm all for it. Your reaction just seems a little over the top to me.

What happened was probably avoidable yes, but that doesn't mean Shaw is the only company who would have had this issue. I think the redundancies and back up plans you expect are not cost effective and slightly unrealistic. I don't think it's fair to expect life to go on without a hiccup when something of this magnitude happens to ANY company.

As for your 9-1-1 comment a small percentage of Shaw phone users didn't have access to the service for a time period. As someone idicated this was told to them and in writing when they got the service, they knew this was a possibility. I bet if you took a poll of all the people who didn't have access to 9-1-1 with Shaw a huge percentage have a cell phone which does have access to 9-1-1.

I haven't been trying to spin your posts, you just give off a tone like you are kind of overreacting and/or dislike Shaw. Is this a serious event? Yes it is, but with people today being so dependant on technology it is to be expected. Is Shaw to blame? Sure they are, and likely will be punished with lost revenue and bills to change their setup so this is less likely to happen again. Do I think that this is as dire as you are making it sound? No.

[Top Shelf]

Some onus has to be on AHS and ATB as well. If you design your infrastructure and applications around a single point of failure (in this case, one physical data center), you open up the possibility of this type of disaster taking you out of business. Is it an acceptable level of risk? Probably it was, as data center catastrophes like this are rare.

I would be asking, why wasn't their a second data center where there applications and servers were hosted out of? I take it there DR plan did not account for total loss of a data center, and while I understand it is extremely expensive and complex, the technology is out there.

Not trying to defend Shaw too much, they are going to take a ton of heat for this, and rightfully so.

[mykalberta]

Originally Posted by Rathji

So I am confused, does Shaw run hosted services for AHS, or it is just the connection to <inserr other service name here> that has been disrupted?

Seems to be claims of a bit of both here.

Hard to know for sure.

What is known is that IBM subleases 3 floors in the Shaw building and they preform contract work for AHS.

Combine that with previous posts about WAN connections being down and it makes sense that Shaw is the ISP of choice for AHS in Calgary (who knows maybe IBM/AHS uses Telus and the only reason they went down is cause power was shut to the Shaw building and hence IBM went down).

The 911 issue, I am less sure of what the issue there might be. I always thought the City ran 911 service so not sure why the Shaw building going down would affect it unless they use Shaw as an ISP as well but you would think that they would use Enmax Envision - in my experience if its available its far more cost effective if you are in Calgary vs Telus and Shaw.

I thought only Shaw customers were without 911 access hence the news report to go to a neighbours house or use your cell to call 911.

[mykalberta]

Originally Posted by Top Shelf

Some onus has to be on AHS and ATB as well. If you design your infrastructure and applications around a single point of failure (in this case, one physical data center), you open up the possibility of this type of disaster taking you out of business. Is it an acceptable level of risk? Probably it was, as data center catastrophes like this are rare.

I would be asking, why wasn't their a second data center where there applications and servers were hosted out of? I take it there DR plan did not account for total loss of a data center, and while I understand it is extremely expensive and complex, the technology is out there.

Not trying to defend Shaw too much, they are going to take a ton of heat for this, and rightfully so.

ATB has said Credit Card transactions at bick/mortar stores were not affected, only logging in online to I assume check balances (maybe also on web transactions).

[Hockeyguy15]

Originally Posted by mykalberta

The 911 issue, I am less sure of what the issue there might be. I always thought the City ran 911 service so not sure why the Shaw building going down would affect it unless they use Shaw as an ISP as well but you would think that they would use Enmax Envision - in my experience if its available its far more cost effective if you are in Calgary vs Telus and Shaw.

I thought only Shaw customers were without 911 access hence the news report to go to a neighbours house or use your cell to call 911.

Yes only some Shaw customers were without 9-1-1. I'm not 100% sure but I don't think 9-1-1 was down, I think it was Shaw phone was down so people didn't have the ability to call 9-1-1.

[MotoRacer]

So many doctors and staff ready to lose it today.

stressed.

don't get much worse, except for natural disaster type event

[Hockeyguy15]

Originally Posted by mykalberta

ATB has said Credit Card transactions at bick/mortar stores were not affected, only logging in online to I assume check balances (maybe also on web transactions).

Debit cards, cedit cards were never affected. I used my debit card at around 4:30 yesterday multiple times without issue.

I guess some direct deposits that were made yesterday were late being put into peoples accounts but that should be the extent of it.

According to twitter everything works except for online banking, their website, and you will not have accurate balances from ABM's. You need to go to a branch if you need an accurate account balance. They also said bill payments, and direct deposits are all back to normal today.

[sclitheroe]

Originally Posted by mykalberta

Hard to know for sure.

What is known is that IBM subleases 3 floors in the Shaw building and they preform contract work for AHS.

And for ATB.

And the failures for both those companies has nothing to do with Shaw - Shaw just leases space, it was those companies and their IT partners that made decisions about where to house gear, and whether or not to spread that gear around multiple datacenter facilities, have a hot standby site, etc.

[mykalberta]

Originally Posted by sclitheroe

And for ATB.

And the failures for both those companies has nothing to do with Shaw - Shaw just leases space, it was those companies and their IT partners that made decisions about where to house gear, and whether or not to spread that gear around multiple datacenter facilities, have a hot standby site, etc.

To be honest ATB actually seemed like essential things are designed for some sort of redundancy. Debit and Credit transactions wernt affected and their brick/mortar stores provide redundancy for checking balances and pay your bills if you did them online.

Its the AHS reports that are of serious concern. Private companies (especially banks) should be of little concern to the average Albertan IMO.

It depends. Its been reported the "buidling" was planned for dual redundancy and yet a mechancial room fire on 1 floor caused the redundancy to mean nothing. If IBM decides to place data servers in a building claiming to have have a redundancy guarantee then its the buildings fault (who owns the building, who knows).

The AHS issues eventually reside with AHS whether or not it was directly their fault, they chose IBM as a contractor because its cheaper than having it house. That is why you outsource anything.

I realize how government works, for AHS only looks after the CHR then it doesnt make financial sense to have multiple data centres. The problem is they dont seem to have 1 proper 1. Offloading that responsibility to a contractor in a 3rd party building is not great planning for a private company and is terrible planning for AHS which runs hospitals. The very least they need to do is have a proper data centre in a building they own. Those would be defendable costs.

Having 2 data centres for AHS if all they handle is Calgary is too cost prohibitive, if they had that and the cost came out then every b1tcher and complainer out there would ask "why arent we spending that money on fronline workers blah blah blah). Even private companies with billions a year in profit only have 1 offsite data centre but that data centre will have network and power redundancies (100% is impossible but 3-6 9's is more likely).

[You Need a Thneed]

Our kids' appointments for shots were cancelled this morning due to this - at Village Square community health centre.

Based on that, I can certainly see why surgeries could be cancelled, too.

[Torture]

My company's been without network (including email, our enterprise one system, etc) and internet access across the country for the past day and a half because of this.

Not much you can do nowadays without internet access - I at least was able to tether my phone and get some work done today. Can't even access my files because they're all saved on the network. Crazy.

[Resolute 14]

Originally Posted by Notorious Honey Badger

Pretty lousy excuse from Shaw. Peoples lives are being affected who aren't even their clients.

Shaw deserves blame, but it does not deserve remotely close to all of the blame, really.

Chances are high that AHS's services were contracted through IBM. So on top of Shaw, you can also blame IBM for not having a backup plan of their own and you can blame AHS itself for failing to have a viable backup plan of their own.

[GP_Matt]

Originally Posted by InglewoodFan

I'm hearing that Alberta registries won't be up before the weekend. Combine this with the fact that I am signing off on house purchase tomorrow and hoping to take possession Monday, does anyone know if I am screwed if the land title registry is still down?

From what I heard Alberta Land Titles Office is currently migrating to their backup servers and are in the testing phases right now. It might be up and running tomorrow. I certainly hope so as our company uses the LTO site pretty extensively.

That said, for a home purchase the lawyer should have the title pulled in advance to check for any relevant encumbrances that might affect the transfer. The actual paperwork usually takes a week or two to process by LTO anyway. Idon't know the whole process as the lawyer takes care of most of it,but I think you should be okay on the possession front.

[DownInFlames]

I didn't know LTO's system was down. That explains why I didn't have any deliveries there today. Stupid Shaw cost me money.