Rogers Services Down Nationwide

[Inglewood Jack]

Originally Posted by Inglewood Jack

very much looking forward to my $5 billing credit for next month

I was overly optimistic, my credit will be $1.90 for July.

[Wormius]

Originally Posted by Inglewood Jack

I was overly optimistic, my credit will be $1.90 for July.

That’s almost exactly 1/30 of my bill (for a $43/mo plan). I guess they’re going with the bare minimum.

[activeStick]

Originally Posted by photon

It could have been a change, just one with a really bad payload. Then when the BGP update starts knocking all the systems offline you don't have access anymore to be able to roll back. And maybe the people with physical access didn't have the necessary level of access to roll back the changes, or something else with their design prevented an easy rollback in that specific failure mode.

I've had to write RCAs for failures that were a perfect storm of unusual circumstances before, though usually isolated to a single system. I agree would love to know the details, I assume that we'll get some level of explanation at some point.

Good on you for writing what sounds to be thorough RCAs. I've seen some laughably terrible RCAs for Sev 1s which are one or two sentences long but nobody cares anymore because the issue has already been resolved and closed.

[JohnnyB]

Originally Posted by Inglewood Jack

I was overly optimistic, my credit will be $1.90 for July.

Canadian telecoms have a long history of getting away with not having to care much about customers. Even an epic screw up isn't likely to change that

[cam_wmh]

Originally Posted by photon

It could have been a change, just one with a really bad payload. Then when the BGP update starts knocking all the systems offline you don't have access anymore to be able to roll back. And maybe the people with physical access didn't have the necessary level of access to roll back the changes, or something else with their design prevented an easy rollback in that specific failure mode.

I've had to write RCAs for failures that were a perfect storm of unusual circumstances before, though usually isolated to a single system. I agree would love to know the details, I assume that we'll get some level of explanation at some point.

Beyond an internal Rogers RCA, I imagine there’ll be clients with muscle (enterprise etc), demanding to see it. Really, due to the scale the public should be privy too.

[cam_wmh]

Another thought, along RCA & fallout.. Rogers 3rd quarter reporting should be interesting, but I’d love to see a post mortem churn report from this outage.

[activeStick]

No doubt there will be tons of eyes at the top levels internally on this issue given the impacts. From what I've seen there's a lot less rigor around these maintenance releases because they include much less complex changes and are limited in size of the change for approval to go as part of a maintenance release which can happen as frequently as every two weeks. But because the changes are fairly simple, it's easy to under-appreciate potential impacts as you automatically assume that there's no need to do the sort of impact assessments that happen if something goes in a major release.

[photon]

Originally Posted by activeStick

Good on you for writing what sounds to be thorough RCAs. I've seen some laughably terrible RCAs for Sev 1s which are one or two sentences long but nobody cares anymore because the issue has already been resolved and closed.

Hah yeah that's kind of how they were taken too. Oh well I write them to the level that I would want for me to understand what happened, I get that less technical leadership ultimately just wants to know a) do we understand what happened and b) can it happen again and if it can c) how do we prevent/mitigate/whatever.

[Jimmy Stang]

nm

[cam_wmh]

Could be a seldom used credit code, that some VP chose for tracking purposes. These codes can be timely to have programmed.

Or I’m just blowing smoke, and it’s completely unrelated haha

[Firebot]

Originally Posted by activeStick

No doubt there will be tons of eyes at the top levels internally on this issue given the impacts. From what I've seen there's a lot less rigor around these maintenance releases because they include much less complex changes and are limited in size of the change for approval to go as part of a maintenance release which can happen as frequently as every two weeks. But because the changes are fairly simple, it's easy to under-appreciate potential impacts as you automatically assume that there's no need to do the sort of impact assessments that happen if something goes in a major release.

These type of software maintenance updates tend to be done by outsourced resources who follow a scrip that can be paid for dimes to a dollar when compared to an in house operator. Rogers infamously completely outsourced their IT department back in 2009, and they also recently changed to a new company for operational stability just a few months ago, undoubtedly somewhere in the transition knowledge transfer and quality suffered, and someone ran a scrip on hardware unfamiliar to them without the experience on troubleshooting or planning.

https://www.businesswire.com/news/ho...onal-Stability

And that's precisely the reason why Rogers doesn't know what occurred when asked, as the update was likely done by a 5$ a day scrip tech in India out of sight and out of mind of the high level executives that patted themselves on the back in new cost cutting IT contracts last year with a big bonus to match.

People only point to IT when there is a problem unfortunately, and when things work, execs go "why do we need IT" and IT is usually the first place to cut or be outsourced.

[timun]

^^The company I work for did the same thing, and the efficiency and reliability of our IT services cratered. Some Chief Technology Officer got a bunch of pats on the back and a big bonus for "saving the company a lot of overhead", but in effect it has wasted more of the everyday (revenue-generating) workers' time.

More and more we're being pushed to delegate work to offshore offices that pay their people a fraction of what the average Canadian employee is paid. Just part of the race to the bottom, all in the name of "staying cost-competitive".

[Firebot]

I have so many stories that it would make people question everything, there's a reason why I sound so pessimistic all the time

This is why when people were hypothesizing things like cyber attacks...I was like no...the answer is actually a lot worse then that.

And the problem is systemic corporate and government internal corruption, it may have happened to Rogers now, but Interac, the defacto financial system in Canada for banking went down as a result of atrocious executive business decisions.

Frankly I think the level of incompetency that occurred with Interac as a result of an outage not related to their own system is 100 times worse then Rogers and all IT executive levels there should be fired. It's networking 101 to have redundancy and they failed one of the most basic tasks of network redundancy (Fido as a backup is not redundancy!)

Our Canadian infrastructure system is built on this frame. Our country is only a bad software update away from a total collapse and we just saw the results.

[Hack&Lube]

Originally Posted by Firebot

These type of software maintenance updates tend to be done by outsourced resources who follow a scrip that can be paid for dimes to a dollar when compared to an in house operator. Rogers infamously completely outsourced their IT department back in 2009, and they also recently changed to a new company for operational stability just a few months ago, undoubtedly somewhere in the transition knowledge transfer and quality suffered, and someone ran a scrip on hardware unfamiliar to them without the experience on troubleshooting or planning.

https://www.businesswire.com/news/ho...onal-Stability

And that's precisely the reason why Rogers doesn't know what occurred when asked, as the update was likely done by a 5$ a day scrip tech in India out of sight and out of mind of the high level executives that patted themselves on the back in new cost cutting IT contracts last year with a big bonus to match.

People only point to IT when there is a problem unfortunately, and when things work, execs go "why do we need IT" and IT is usually the first place to cut or be outsourced.

This is the problem when you prioritize profits for the Rogers family and shareholders over everything else.

You are right, executives in the past have definitely received bonuses for reducing operating and capital costs by cutting back in IT.

What's stupid is IT is the backbone of their entire business. This is akin to going to hospital and them telling you they are cost-cutting you to cheaper surgeons and giving you a discount pacemaker made in the 3rd world so the hospital can have more profits.

The analogy is apt because of the failure of the system affecting 911 services and costing billions to the economy.

Telecom/internet needs to be federally regulated as an essential utility and enormous fines need to be legislated for any outage to force companies to maintain investment into technology, processes, and safeguards.

Nobody likes doing the routine IT maintenance because it becomes routine, management prefers to sleep while people labor after hours, and the ones told to do it feel isolated, underbudgeted, alone, and scapegoated if things go sideways.

[Azure]

At what point should independent oversight dictate how a financial system like Interac should be operating in terms of backup capabilities, resilience against cyber attacks, network outages, etc?

Should we not consider Interac as an essential service?
Banking, ATMs, credit card processing, hospital pagers, 911 service?

Like I don't get how we even got here.

To me this is a complete failure of policy & systems designed to prevent something like this from even being able to happen. What is the point of a SLA with Rogers right now?

[Fuzz]

I find some of this ragey ragey pretty funny. Sure, Interac ####ed up by not having a backup. But it's not like there aren't other ways to pay for things. Maybe keep some cash with you, as your own personal responsibility back up? Or does the government need to pull up your boot straps for you?

Before we had cell phones people may not have ever noticed their phone service went out. If a day of no cell service is this frustrating to your life, maybe it's time to examine your dependence on it.

And yes, I know business and hospitals and all that stuff, but it's technology, sometimes things will happen. Maybe a solar flair blasts everything away one day. Will you freak out that government needs to make everything resilient to super volcanoes? We should probably build more resilience into the critical systems we need, and worry less about not being able to text aunt Becky.

If you really really need service for your business, setup with a separate provider. Hell, we did that years ago at our small company, and despite my hatred fo Telus, pay them for the backup. And we don't even NEED it. Yet so many big companies get caught with their pants down? That's more a them problem than a Rogers problem. We should expect these situations.

[Wormius]

I think we need is a provision where if your primary network provider craps out, you automatically roam to an available network at the expense of your primary provider. The CRTC should be pushing this.

[Firebot]

Originally Posted by Wormius

I think we need is a provision where if your primary network provider craps out, you automatically roam to an available network at the expense of your primary provider. The CRTC should be pushing this.

You understand the nature of the Rogers outage...right?

You can't roam...if you don't exist to be able to roam (as the people who were in the states painfully found out)

[Swift]

5 days credit - https://www.cbc.ca/news/business/rog...tage-1.6518545

Better than the 1 day I expected. TBH I wasn't particularly affected by the outage as only my cell phone is with Rogers (I was able to use Shaw for wifi) so 5 days is pretty decent for me.

[FlameOn]

CRTC gives Rogers 10 days to provide root cause of the problem and a mitigation plan for possible future occurrences.

https://mobilesyrup.com/2022/07/12/c...swer-questions