About that Rogers Outage (Update)

Well. I was proven wrong on the “we may never know” argument as presented in my last post, from a few hours ago….

So what went wrong? We don’t yet know, and, we may never know. Rogers may never know for sure.

As of10:34 PM about 8 hours after I wrote that, Rogers has released the following:

..coding from the update deleted a routing filter that “allowed for all possible routes to the Internet to pass through the routers,” which flooded and overwhelmed the core network, causing it to stop processing internet traffic altogether.

“As a result, the Rogers network lost connectivity to the Internet.”

Holy fuck balls.

Any organizations IT infrastructure is only as strong as the partner vendors they rely on. For customers that rely on Rogers, they were let down. Rogers relies on network hardware vendors, and…

They were massively let down.

I cannot express how bad this is.

One of the stated reasons Huawei was banned as a 5G vendor was for its buggy sloppy code, that did not meet “Western Standards”

Even if Huawei cooperates unreservedly, bringing its vulnerable, buggy firmware up to Western standards.

“There were security and quality issues that were rampant throughout that code…Whoever’s going to license that technology needs to know they have a lot of work ahead of them to really remediate those issues.”

Their vendors are supposedly trusted partners withcodes up to “Western Standards.” They really shit the bed on this one. An update that deletes key configuration entries? An update that overrides the design that customers put into their devices? This is sloppy. This is vulnerable. This. Is. Why. Huawei. Was. Considered A Security and Stability Threat!!!

Why wasn’t this detected earlier? Well, as I put in my last post there is no “Internet in a lab” to test against. This update contained a bug that was found by connecting the Rogers network to the Internet.

Customers were let down by a partner, a partner many of us, including big companies like Interac, trusted. That partner was Rogers.

Rogers was let down by a partner, a partner they trusted and be the future of their company on.

This is bad. Rogers and Telcos worldwide now have to face the fact that they can’t trust Ericsson to deliver quality code. That has implications. Expensive implications.

UPDATER: Reading the Rogers submission to the CRTC closely, there is a possibility that Rogers made the error, and not the vendor. Their detailed response, with critical redactions (unfortunately) is listed here: General Information — Documents received from Companies: 8000-C12-C12–202203868 | CRTC at document “2022–07–22 — Rogers Communications Canada Inc.”

If the Rogers network team made the errors, then that is quite obviously a different situation than detailed above. As the detailed root cause is redacted, I can’t call it one way or the other.

But it may have been this:

If it was. Well. “Holy ‘Effin Shitballs Batman! That’s amateur hour! Professional Telecommunications company, my ass!”