Rogers Services Down Nationwide

[dammage79]

It's insanity to me that the primary transaction utility in the country is solely sourced through Roger's. How the hell does that happen? I wouldn't give two sheets if Roger's went down forever if Interac didn't solely go through them.


God I hate Roger's. They make it so easy to.

[1_Flames_Fan]

Originally Posted by dammage79

It's insanity to me that the primary transaction utility in the country is solely sourced through Roger's. How the hell does that happen? I wouldn't give two sheets if Roger's went down forever if Interac didn't solely go through them.


God I hate Roger's. They make it so easy to.

You would think something as important to the economy as Interac would have a backup or redundancy in place but apparently not.

[djsFlames]

Rogers is the worst tech company of all time.

[djsFlames]

Stampede day 1 would be massively affected too.

[GreenLantern2814]

Originally Posted by Nancy

I have seen the following posted which believes it was human error:

https://blog.cloudflare.com/cloudfla...age-in-canada/

Also, as of a few moments ago, about 17% of their traffic has been restored. I have service right now.

https://twitter.com/user/status/1545591714227933186

That time lapse video of their network collapse looks like a brain dying.

Human error, if the error is some exec clicked something they shouldn’t have, gave someone who knew what they were doing admin access, who then set up the conditions to take down the entire Rogers network.

That can’t happen by accident.

[Wormius]

A Rogers exec doesn’t have the kind of ability to shut down the network like that. It would more likely be some developer that ended up triggering something with an update of some kind.

[MRCboicgy]

I noticed my tweet on page 2 was deleted - the poster posited that this could have been caused by a software update pushed out by someone like Ericsson to the infrastructure. Guessing that wasn't right.

[cam_wmh]

Originally Posted by GreenLantern2814

I work for a company called Apex Wireless.

Happy to help anyone who needs it, send a PM for more info.

Apex is bigger than Tom Harris? All the VARs'? Partners? Not so sure.

Originally Posted by GreenLantern2814

The phones and plans I sell are all on the Telus network - I primarily deal with SMB clients, but with this service outage, today the net is cast a little wider.

My personal cell is with Rogers, it’s functionally a brick right now - colleagues on Telus are not so impacted, though obviously Rogers going down nationwide has many, many ripple effects.

Telus is the strongest network in Alberta/BC.

Rogers has the best distribution of towers coast to coast, but they’re strongest in Ontario/central Canada.

Bell has the strongest network in Quebec and the Maritimes.

There’s clearly no sharing of towers.

Curious how you conclude there's no sharing of towers ('radios'), where TELUS & BELL's do exactly that, for a large segment of each's incumbent regions.

That said, I strongly advocate any competent IT Mgr/Director, to research TelCo competency via international carrier analytical companies, like OpenSignal, & Teragence, akin to referring to Gardner for most other solutions. Obviously, carrier redundancy is a no brainer, right to the backhaul.

[cam_wmh]

Originally Posted by Erick Estrada

I'm no lawyer but will some businesses be able to sue Rogers for lost revenue?

Within the business/commercial spaces - generally the large
/enterprise market segments, SLA’s and guarantees are written into the agreements, with obviously a higher yield to the carrier —— until outage occurs.


Also, forgot to add on my last post, and I mean it’s quite elementary.

Simply identify revenue to network investment, as a percentage.

Of the Big3, one is single digits, the other two in the teens. Nobody today, is guessing who the former is.

[Firebot]

Originally Posted by GreenLantern2814

That time lapse video of their network collapse looks like a brain dying.

Human error, if the error is some exec clicked something they shouldn’t have, gave someone who knew what they were doing admin access, who then set up the conditions to take down the entire Rogers network.

That can’t happen by accident.

Of course it can.

But accidents are usually a result of negligence, hardware failure, incompetence or someone deciding to cut corners one too many times or poor company policies finally having a side effect. I've seen it many many times in my field. Some of the biggest outages were caused by an update not properly tested, or unauthorized change on a server by one person.

That's why most major companies follow strict change management processes, but even the most strict change management process doesn't prevent that single lazy off shift worker to skip a few steps to go home early and may have caused a failure on critical infrastructure that would normally be routine.

[afc wimbledon]

Originally Posted by dammage79

It's insanity to me that the primary transaction utility in the country is solely sourced through Roger's. How the hell does that happen? I wouldn't give two sheets if Roger's went down forever if Interac didn't solely go through them.


God I hate Roger's. They make it so easy to.

You would think at the very least an all purpose message should have been sent by all providers so at least those in range of wifi would know, I was fine with losing the phone but the trip to costco was a waste of time I could have done without and judging by the language I wasnt the only one, a text message would have prevented it

[FlamesAddiction]

https://www.msn.com/en-ca/news/canad...d=winp2sv1plus

I don't know how to imbed, but that looks like a man who shat himself.

[Wormius]

Originally Posted by afc wimbledon

You would think at the very least an all purpose message should have been sent by all providers so at least those in range of wifi would know, I was fine with losing the phone but the trip to costco was a waste of time I could have done without and judging by the language I wasnt the only one, a text message would have prevented it

Costco was still taking Mastercard, at least in Calgary.

[GreenLantern2814]

Originally Posted by cam_wmh

Apex is bigger than Tom Harris? All the VARs'? Partners? Not so sure.




Curious how you conclude there's no sharing of towers ('radios'), where TELUS & BELL's do exactly that, for a large segment of each's incumbent regions.

That said, I strongly advocate any competent IT Mgr/Director, to research TelCo competency via international carrier analytical companies, like OpenSignal, & Teragence, akin to referring to Gardner for most other solutions. Obviously, carrier redundancy is a no brainer, right to the backhaul.

I didn’t say largest, I said “only national partner” - We have clients across the country with offices in BC, Alberta and Ontario. Other dealers may have more people, my understanding is that we have operations in more places than other dealers. If that’s incorrect, I’ll bring it up at the next meeting.

Re : tower sharing. It was a long day, at the end of a stressful day, and I’m new.

clearly my comments are incomplete, if not indeed entirely incorrect.

[bossy22]

Originally Posted by Wormius

Costco was still taking Mastercard, at least in Calgary.

My Mastercard failed at Costco yesterday. It's with BMO. Don't know if that had anything to do with it. Fortunately I just sold a set of road wheels so I had a bunch of cash in my wallet.

[Azure]

Aren't we glad that the CRTC makes sure things like this don't happen?

I'm wondering who the moron is that decided that critical infrastructure such as Interac shouldn't have a failover. Also, why isn't it mandated when something like this happens, everyone on Rogers can automatically failover on the Telus or Bell Network?

Even hard that pagers and what not in hospitals were affected.

[Azure]

https://twitter.com/user/status/1545448385070731264

[Erick Estrada]

Originally Posted by Azure

Aren't we glad that the CRTC makes sure things like this don't happen?

I'm wondering who the moron is that decided that critical infrastructure such as Interac shouldn't have a failover. Also, why isn't it mandated when something like this happens, everyone on Rogers can automatically failover on the Telus or Bell Network?

Even hard that pagers and what not in hospitals were affected.

Positive out of this is that it has exposed Canada's cellular infrastructure as and hopefully kickstarts investigations into the Shaw acquisition.

[Timbo]

Y2K 22 years and 7 months late.

[Azure]

Originally Posted by Erick Estrada

Positive out of this is that it has exposed Canada's cellular infrastructure as and hopefully kickstarts investigations into the Shaw acquisition.

Yawn.

Nothing will change. Canadians have shown they don't give a #### about critical issues. We elect governments back into power regardless of their incompetence and corrupt ties to big business.

Next week the Trudeau government, CRTC officials and Rogers executives will have a fancy steak dinner at taxpayer expense, and the week after Rogers will say they are investigating, the week after that their investigation report will say they did nothing wrong, and the week after that the Trudeau Government, CRTC will give Rogers a bit ol' pat on the back and brag about how they are the gold standard of the Canadian corporate world, and how Canadians should be thankful we have freedom and the priviledge of having a cell phone, Interac and cable television, and the high prices are needed to protect freedom and liberty for all.

A few weeks after that everything will be much ado about nothing, and the Trudeau government will actively block any investigation into anything that will make Rogers look bad.