Shaw Court on fire

[psicodude]

As far as I can tell with the limited amount of information there is out there, the only thing Shaw really did wrong was have their primary and backup generators too close to their transformers, thus allowing the sprinkler system to bring down everything. My guess is that they took a look at the cost of moving some of this equipment around versus the chances of an event like yesterday happening and decided the risk was not worth the cost. this type of thing happens daily all over the world. Unfortunately, they got burned.

As others have already stated though, a lot of blame needs to go to the customers that chose to co-locate their servers at Shaw (or IBM). That building is in a horrible location for a major data centre and the fact that it is also a business/retail space makes it even worse. Again, my guess is that AHS saved a few bucks by hosting with Shaw and it back fired. It sucks, but it happens. Anyone recall Instagram being down last week?

[Slava]

So at this point you can't register a vehicle, right? I'm not sure what to do. I picked up a trailer (although want to do that physically tomorrow) and have no plate. Do I just hook it up and produce a bill of sale if I get pulled over? I bought it privately, so I need to get it off the guys property...maybe I'm asking in the wrong thread. It's only sort of related!

[photon]

Originally Posted by Top Shelf

It sucks, but it happens. Anyone recall Instagram being down last week?

Yeah, that was Amazon's EC2.. first they had a cable fault at a utility cause a power outage.. then after 9 minutes on backup generators one of them failed due to a faulty cooling fan. Then after four minutes the backup backup generator's circuit breaker opened because it was incorrectly configured.

[kermitology]

Originally Posted by psicodude

As far as I can tell with the limited amount of information there is out there, the only thing Shaw really did wrong was have their primary and backup generators too close to their transformers, thus allowing the sprinkler system to bring down everything. My guess is that they took a look at the cost of moving some of this equipment around versus the chances of an event like yesterday happening and decided the risk was not worth the cost. this type of thing happens daily all over the world. Unfortunately, they got burned.

As others have already stated though, a lot of blame needs to go to the customers that chose to co-locate their servers at Shaw (or IBM). That building is in a horrible location for a major data centre and the fact that it is also a business/retail space makes it even worse. Again, my guess is that AHS saved a few bucks by hosting with Shaw and it back fired. It sucks, but it happens. Anyone recall Instagram being down last week?

I wouldn't call it a major data centre. Only three floors are used for data storage. The rest is office space, studios and retail.

The data centre at the campus on the other hand..

[psicodude]

Originally Posted by kermitology

I wouldn't call it a major data centre. Only three floors are used for data storage. The rest is office space, studios and retail.

The data centre at the campus on the other hand..

Oh, absolutely. Which is why I believe AHS (or IBM) chose to host there....it's probably cheaper than Q9 or Care Factor.

I haven't seen the new building at the campus since there were just footings there, but the place is going to be incredible. Getting a bit off topic here, but does anyone know if they are going to offer hosting as a product out of there, or is it strictly Shaw in-house stuff?

[Bill Bumface]

I love seeing people who outsourced to IBM getting burned (ignoring the cancelled surgeries part). Continue to try convince yourselves it was a good idea....

[InglewoodFan]

Originally Posted by GP_Matt

From what I heard Alberta Land Titles Office is currently migrating to their backup servers and are in the testing phases right now. It might be up and running tomorrow. I certainly hope so as our company uses the LTO site pretty extensively.

That said, for a home purchase the lawyer should have the title pulled in advance to check for any relevant encumbrances that might affect the transfer. The actual paperwork usually takes a week or two to process by LTO anyway. Idon't know the whole process as the lawyer takes care of most of it,but I think you should be okay on the possession front.

Thanks, I just got back from the lawyer. I should get possession on Monday no matter what, but if LTO isn't online I have to either buy title insurance or pay the interest penalty to the seller until the title can be registered. Few hundred bucks. Lawyers are scrambling, lots of deals this week have this problem.

[Resolute 14]

Originally Posted by hulkrogan

I love seeing people who outsourced to IBM getting burned (ignoring the cancelled surgeries part). Continue to try convince yourselves it was a good idea....

It goes both ways, really. On the one hand, this does suck. On the other, if any of our SAP servers have an issue, our solution is "Dear IBM, you have not much time left to fix the problem, per our contract" and walk away.

Honestly, the odds of this happening once were remote. And yeah, we've had to scramble the last couple days, the fact is, we'll be getting fees back from IBM because they failed to deliver on the service.

[onetwo_threefour]

I was interviewed on Global News today about the situation vis-a-vis Land Titles Services. Short answer is that all the lawyers I've had to deal with so far, including our office, are allowing key release on tenancy at will when acting for Seller, but on the Buyer's side we're not releasing money unless we happen to have title insurance from a provider who allows us to use a search done prior to the closing date. The Buyer has to pay a bit of interest to the seller, but for the most part it's saved on their mortgage, so the only real effect is that Sellers are getting paid a couple of days late. I had one Seller who is moving to Nova Scotia and couldn't wait for money so we had to scramble to organize bridge financing for them, but other than that type of thing, it's being handled pretty responsibly by real estate firms IMO and it should end up being fairly minor in the vast majority of cases.

See the interview: I don't think I can embed from Global's site...

http://www.globaltvcalgary.com/video...p=1&s=dd#video

[photon]

Any word if it'll be cleared up soon? I've got a deal closing in 8 days.

[GP_Matt]

I was told yesterday that LTO has switched to their backup system and are currently testing it now. There was no timeline but I expect it to be up on Monday.
I hear they are allowing some transactions in person but can't remember which ones.

[onetwo_threefour]

One of my assistants has a friend working at LTO and, off the record, we're hearing it may not be up before Tuesday. I mentioned this to Tony Tighe when I met with him this afternoon and he had already spoken to a rep from the government services minister's office and was advised that their feeling is Monday. Based on that info and the fact that a registry office owned by an acquaintance of mine is closing down for the whole weekend (usually open 7 days a week), I suspect it will be at least Monday. Having said that, I suspect that your closing is far enough out that it won't be impacted by this photon.

[onetwo_threefour]

Originally Posted by GP_Matt

I was told yesterday that LTO has switched to their backup system and are currently testing it now. There was no timeline but I expect it to be up on Monday.
I hear they are allowing some transactions in person but can't remember which ones.

I hadn't heard that but probably builders liens would be my guess...

[stuck_in_chuk]

The outages to AHS did have more of an impact than just requiring people to fill out a form for a hospital appointment - as mentioned, it did slow down an already overloaded system, and all the paperwork acquired over the past few days will have to be entered into the computer system at some point. I know that 2 of my colleagues are working all day today to enter in our department paperwork over the past few days. Thankfully, we work in a department that doesn't operate 24-7.

I work in the radiopharmacy at Foothills. Our impact was minimal, although we did have to calculate how to make our radioactive preps by hand rather than by computer (which added extra time and potential for error to the process). Our inventory system was also offline, so we had to keep track of incoming and outgoing stuff by hand as well.

However, the biggest problem we saw was with getting laboratory results. Normally, we could go to the computer to check lab results before doing a test. We spent the better part of yesterday morning trying to ensure that a patient had a negative pregnancy test. I'm sure that other parts of the hospital faced even worse challenges.

[onetwo_threefour]

Just an update for those curious about Land Titles. There are now some conflicting reports that say that LTO may not be back up until Thursday. Service Alberta is saying nothing at all, the update on their website from yesterday was pretty vague.

At least one lender thinks Land Titles wil be back up today, but I don't think they have any better info than the rest of us.

[GP_Matt]

The LTO website is really frustrating. Just a note that says they are aware of the problem and hope to have it fixed soon.

Would it be that hard to keep us informed so that arrangements can be made.

[onetwo_threefour]

For what it's worth, just heard from a second source that LTO is not likely to be up before Thursday.

[Bill Bumface]

Originally Posted by Resolute 14

It goes both ways, really. On the one hand, this does suck. On the other, if any of our SAP servers have an issue, our solution is "Dear IBM, you have not much time left to fix the problem, per our contract" and walk away.

Honestly, the odds of this happening once were remote. And yeah, we've had to scramble the last couple days, the fact is, we'll be getting fees back from IBM because they failed to deliver on the service.

Yeah, after the guy you had to hire after the outsourcing for $300,000 a year finishes the impending contract dispute over the next several months.

I really love these setups.

Pre-outsourcing:

Client: "I need you to grab X numbers for me on this system"
Me: "ok"
*45 minutes later, numbers prepared and sent*

Post-outsourcing:

Client: "I need you to grab X numbers for me on this system"
Me: "I don't have a project code to charge this against, you'll have to talk to your contract liaison"

*3 weeks of contract dispute and 45 minutes later, numbers prepared and sent, 4 hours of special consulting charged to client*

[old-fart]

Bottom line is that the IBM datacenter in Shaw Court is, at the very best, a Tier 2 facility that has known power problems in the past, and will continue to have power problems. Certain services should NOT be in that facility unless there is a proven DR solution in place - a hard lesson that was forced on many businesses and services last week (and continues to be felt this week).

Hopefully that lesson was actually absorbed, and changes can be made.

[InglewoodFan]

Epilogue to my story, I got my keys today. I bought title insurance, thanks government of alberta for giving me something else to spend money on while purchasing a house. I apparently have lots of it.