Quote:
Originally Posted by Slava
Well the sensationalism is definitely a factor and I probably fell victim to that to an extent. When I heard on the radio that they advised people to have their neighbour call 911 for them if they needed it though, I had to laugh.
|
Counterintuitive as it is, the news report was helpful. If you are a Shaw customer with an outage, but your neighbour is on Telus, they may be able to get through.
As far as redundant backups go, as I said, I would bet AHS does have alternatives for critical systems. But from an IT perspective, this was a catastrophic event and right close to being the worst case scenario (namely, building burning down).
From my company's perspective, we have full tape backups of every server we have at Shaw and those tapes are stored at a third location. While we hope our systems will recover with only a little bit of massaging (we already know the hardware is likely fine, it's the data integrity we are concerned with), we are also sourcing additional servers at our other data centre and will restore from those backup tapes if necessary. In our case, we should be fully recovered from a near-worst-case-scenario in less than 36 hours.
From Shaw's perspective, I can only speculate: Shaw Court was not built by Shaw, and was not originally intended to be a data centre. In fact, only a few floors (3-5, iirc) serve that role. the remaining floors are office space. For that reason, they simply won't have the kind of fire suppression systems that either Q9 or their new data centre will. Nor would it be logical for several reasons - namely cost and the rapid death of any poor office worker who doesn't get out fast enough.
Shaw probably felt they had things covered with their backup system. As you noted though, poor engineering of that backup was a pretty major screw up. But, given you can't really flood your building to test, I suspect they thought they were covered for this type of emergency. Is anybody really surprised they failed to anticipate the fire suppression system knocking out the backup generator? Regardless, it is going to cost Shaw millions. I suspect a lot of other data centres across Canada and North America are going to look at what happened here and reevaluate their own set ups.