Rogers Services Down Nationwide

[calculoso]

Originally Posted by Azure

Aren't we glad that the CRTC makes sure things like this don't happen?

Aren’t we glad that cell and internet service are through a business and not government so Trudeau can’t f it up?

Give me a break. Bleat on about small government, government staying out of people’s lives, then blame government when a company screws up? What a world to live in.

[Bill Bumface]

Originally Posted by Monahammer

Today will be a day where MANY corporate policies are updated to have backup telecom options.

I worked somewhere that had two dedicated connections between their datacentres, and they were through different providers using different backbones.

Yet we still had a catastrophic outage. How? Despite having largely different routes, they shared a common rail crossing, and someone dinged both fibre lines with an excavator.

[FlameOn]

Originally Posted by Azure

Yawn.

Nothing will change. Canadians have shown they don't give a #### about critical issues. We elect governments back into power regardless of their incompetence and corrupt ties to big business.


Next week the Trudeau government, CRTC officials and Rogers executives will have a fancy steak dinner at taxpayer expense, and the week after Rogers will say they are investigating, the week after that their investigation report will say they did nothing wrong, and the week after that the Trudeau Government, CRTC will give Rogers a bit ol' pat on the back and brag about how they are the gold standard of the Canadian corporate world, and how Canadians should be thankful we have freedom and the priviledge of having a cell phone, Interac and cable television, and the high prices are needed to protect freedom and liberty for all.

A few weeks after that everything will be much ado about nothing, and the Trudeau government will actively block any investigation into anything that will make Rogers look bad.

Its not that we don't care. What frustrates me immensely is that I don't realistically have a choice on this particular matter in the list of parties that are available when mixed in with other issues. Say for example I want an accountable government, that is pro expansion of public healthcare funding, anti-monopolistic, but fiscally conservative without being socially conservative. That' doesn't exist in this country cuz of how our parties and electoral system works atm. I'm stuck picking my poison.

In this case I'd want steps taken to bring down both broadband + cell service pricing to be in line with at Australia (which has lower population density, and similar cost to serve yet we are 2.5x higher in pricing) while improving current network resiliency. We won't get that with either of our two party choices.

[Azure]

Originally Posted by calculoso

Aren’t we glad that cell and internet service are through a business and not government so Trudeau can’t f it up?

Give me a break. Bleat on about small government, government staying out of people’s lives, then blame government when a company screws up? What a world to live in.

We need independent regulation & oversight of the telecom sector. That is what the CRTC SHOULD be doing. Unfortunately they are just a corrupt entity for government & big business to work together to screw everyone over.

Everyone is now saying 'oh this shows why we need more competition, why we need independent oversight, etc, etc'...and those of us with a shred of common sense are sitting here wondering where the hell everyone has been the last 10+ years we've been complaining about the corruption and stupidity at the CRTC.

Seriously, who is stupid enough to run Interac on a single network without an alternative provider as a backup?

[Azure]

Originally Posted by FlameOn

Its not that we don't care. What frustrates me immensely is that I don't realistically have a choice on this particular matter in the list of parties that are available when mixed in with other issues. Say for example I want an accountable government, that is pro expansion of public healthcare funding, anti-monopolistic, but fiscally conservative without being socially conservative. That' doesn't exist in this country cuz of how our parties and electoral system works atm. I'm stuck picking my poison.

In this case I'd want steps taken to bring down both broadband + cell service pricing to be in line with at Australia (which has lower population density, and similar cost to serve yet we are 2.5x higher in pricing) while improving current network resiliency. We won't get that with either of our two party choices.

All true.

However I still think if Canadians actually spoke to their MPs and demanded action, something would happen. But, most don't care or really understand what is going on.

Look at what is happening in the US regarding Starlink, Dish and that stupid spectrum BS that Dish is trying to pull. The FCC has gotten 90,000+ comments from Starlink users who are pissed off with what is going on.

Canadians will never band together and organize something like that.

[Erick Estrada]

It is weird how Canadians just don’t push back like citizens of other countries when it comes to the government allowing cartels to run consumer industries. I didn’t support the convoy but part of me is a little happy that at least some Canadians have a little backbone and are willing to stand up for something whether I agree with it or not.

[Macman]

It appears that most of the service has been restored.

https://www.cbc.ca/news/business/rog...wifi-1.6514373

[Azure]

Originally Posted by Macman

It appears that most of the service has been restored.

https://www.cbc.ca/news/business/rog...wifi-1.6514373

"We don't understand how the different levels of redundancy that we build across the network coast to coast have not worked," said Kye Prigg, Rogers' senior vice-president of access networks and operations, on CBC's Power & Politics.

Bahahaha. What a mess.

[cam_wmh]

Originally Posted by Azure

We need independent regulation & oversight of the telecom sector. That is what the CRTC SHOULD be doing. Unfortunately they are just a corrupt entity for government & big business to work together to screw everyone over.

Everyone is now saying 'oh this shows why we need more competition, why we need independent oversight, etc, etc'...and those of us with a shred of common sense are sitting here wondering where the hell everyone has been the last 10+ years we've been complaining about the corruption and stupidity at the CRTC.

Seriously, who is stupid enough to run Interac on a single network without an alternative provider as a backup?

The CRTC does a decent job, especially compared to the FCC.

Market competitiveness;
* Multiple new entrant low cost frequency bid options
* MTS sale, a forced split

Ensured Revenue
* Stone-walling the option for carriers to transition to Trusts

Consumer protections
* Removing the 3rd year, as a device amortization option
* Forced shared carrier roaming
* Mitigating data consumption, and roaming overages
* Clear & concise billing
* All devices to be Unlocked

And a couple more aged examples;
* Wireless Number Portability
* Foreborne vs regulated pricing, (CLEC vs ILEC)

[GreenLantern2814]

Originally Posted by Azure

Yawn.

Nothing will change. Canadians have shown they don't give a #### about critical issues. We elect governments back into power regardless of their incompetence and corrupt ties to big business.

Next week the Trudeau government, CRTC officials and Rogers executives will have a fancy steak dinner at taxpayer expense, and the week after Rogers will say they are investigating, the week after that their investigation report will say they did nothing wrong, and the week after that the Trudeau Government, CRTC will give Rogers a bit ol' pat on the back and brag about how they are the gold standard of the Canadian corporate world, and how Canadians should be thankful we have freedom and the priviledge of having a cell phone, Interac and cable television, and the high prices are needed to protect freedom and liberty for all.

A few weeks after that everything will be much ado about nothing, and the Trudeau government will actively block any investigation into anything that will make Rogers look bad.

Sounds like a good election issue for the conservatives to run on.

Be the anti-Inflation, anti-monopoly party instead of the anti-vax/anti-abortion one, and they could win.

Alas.

Originally Posted by Azure

Bahahaha. What a mess.

Could this possibly be because it was an attack?

I’m open to it not being one, and I certainly hope it wasn’t, but could someone who understands this stuff better please help me understand why it wasn’t?

[JohnnyB]

Originally Posted by GreenLantern2814

Could this possibly be because it was an attack?

I’m open to it not being one, and I certainly hope it wasn’t, but could someone who understands this stuff better please help me understand why it wasn’t?

This doesn't explain on what basis the government is willing to provide assurance that it was not a cyber attack, but I kind of suspect that if they were willing to provide assurances they have some info about how this was an internal issue at Rogers.

Rogers outage doesn't appear to be a cyberattack: government official

[SuperMatt18]

Originally Posted by Azure

Seriously, who is stupid enough to run Interac on a single network without an alternative provider as a backup?

This is more Interac’s problem than Rogers to be honest.

I’m sure if Interac wanted to pay Bell or TELUS they would be happy to act as alternative provider / backup.

[GreenLantern2814]

Originally Posted by JohnnyB

This doesn't explain on what basis the government is willing to provide assurance that it was not a cyber attack, but I kind of suspect that if they were willing to provide assurances they have some info about how this was an internal issue at Rogers.

Rogers outage doesn't appear to be a cyberattack: government official

Maybe. They could also know exactly what happened and plead incompetence due to the outrage that would surely follow if it came out that Russia or China had done this deliberately.

I really do hope it’s garden variety incompetence that caused this, but if Rogers still doesn’t know how all their redundancies failed, I have to wonder if it’s because someone turned them off.

Of course, I don’t know anything about how this sorcery actually works - it could be as simple as “this is how an RBMK reactor core explodes”.

I honestly don’t know which is more disturbing.

[JohnnyB]

Originally Posted by GreenLantern2814

Maybe. They could also know exactly what happened and plead incompetence due to the outrage that would surely follow if it came out that Russia or China had done this deliberately.

I really do hope it’s garden variety incompetence that caused this, but if Rogers still doesn’t know how all their redundancies failed, I have to wonder if it’s because someone turned them off.

Of course, I don’t know anything about how this sorcery actually works - it could be as simple as “this is how an RBMK reactor core explodes”.

I honestly don’t know which is more disturbing.

It's probably just an internal screw up. A cyber attack may be a possibility, but no need for stretching towards that conclusion without some evidence. Without evidence it just seems like paranoia.

[Wormius]

Originally Posted by JohnnyB

It's probably just an internal screw up. A cyber attack may be a possibility, but no need for stretching towards that conclusion without some evidence. Without evidence it just seems like paranoia.

Yeah. These kinds of screw ups happen all the time, but generally goes unnoticed because it’s not the same scale.

[dissentowner]

Originally Posted by Macman

It appears that most of the service has been restored.

https://www.cbc.ca/news/business/rog...wifi-1.6514373

Not in SW Ontario. Cell service is back but internet and tv is still down for most.

[OldDutch]

Originally Posted by Wormius

Yeah. These kinds of screw ups happen all the time, but generally goes unnoticed because it’s not the same scale.

Someone sweet talked the CAB chair with “we do this all the time don’t worry”

[Hemi-Cuda]

Originally Posted by OldDutch

Someone sweet talked the CAB chair with “we do this all the time don’t worry”

Unless Roger's change process is on the level of a mom & pop shop, this wasn't a change. Even minor changes need to have some kind of quick rollback planning, the fact that their network is still broke means someone did something outside any approved change window. I'd love to be a fly on the wall for their incident resolution on this one. For something to bring down all of their internet, cell, and TV nationwide it must have been someone ####ing with a core network service. And at that level of critical infrastructure, it was either a senior tech ####ing it up or someone who shouldn't have had that level of access, which is worse

[photon]

It could have been a change, just one with a really bad payload. Then when the BGP update starts knocking all the systems offline you don't have access anymore to be able to roll back. And maybe the people with physical access didn't have the necessary level of access to roll back the changes, or something else with their design prevented an easy rollback in that specific failure mode.

I've had to write RCAs for failures that were a perfect storm of unusual circumstances before, though usually isolated to a single system. I agree would love to know the details, I assume that we'll get some level of explanation at some point.

[Nancy]

The CEO of Rogers said that they believe the outage was the result of a maintenance update. I have not seen any details yet beyond that.

https://www.thestar.com/business/202...er-outage.html