Global Microsoft outage hits airlines, banks and businesses

[Fuzz]

Originally Posted by nfotiu

How did an update like this hit every server worldwide at the same time?

If this is true, it's the usual foe. Incompetence.

What happened here was they pushed a new kernel driver out to every client without authorization to fix an issue with slowness and latency that was in the previous Falcon sensor product. They have a staging system which is supposed to give clients control over this but they pissed over everyone's staging and rules and just pushed this to production.

https://news.ycombinator.com/item?id=41003390

[KelVarnsen]

How very Rogers of them

[Blaster86]

That's a whoopsie-oopsie.

[Reggie Dunlop]

[Firebot]

Originally Posted by Fuzz

If this is true, it's the usual foe. Incompetence.



https://news.ycombinator.com/item?id=41003390

"Who needs change mangement and processes when I want to go on vacation early? Just push it no one will notice"

CIO will likely be forced to resign, in the end processes and security measures are built from the top down.

[Fuzz]

Originally Posted by Firebot

"Who needs change mangement and processes when I want to go on vacation early? Just push it no one will notice"

CIO will likely be forced to resign, in the end processes and security measures are built from the top down.

The CEO won't be racing his car today, that's for sure.


https://www.theautopian.com/think-yo...-this-weekend/

[nik-]

Originally Posted by Fuzz

The CEO won't be racing his car today, that's for sure.


https://www.theautopian.com/think-yo...-this-weekend/

What a dumb article lol. CEO has hobby and he isn't doing it today. Ok?

[Fuzz]

Originally Posted by nik-

What a dumb article lol. CEO has hobby and he isn't doing it today. Ok?

It's a car site, they need to leverage the connection to a massive news story somehow. Doin' it for the clicks!

[Erick Estrada]

LOL I take the afternoon off today to do estate stuff getting checks to pay charities, beneficiaries, etc and of course I walk into TD bank and they are down. Oh well it's a nice afternoon to go do something else I guess.

[activeStick]

Originally Posted by nfotiu

How did an update like this hit every server worldwide at the same time?

I'm guessing some genius decided his code change was good and pushed it direct to prod

[Firebot]

It's absolutely mind boggling just how massive of an disaster this is. This will take some enterprises weeks to figure out and fix in some instances and in some cases may not be possible. You can't send updates to a PC that is in a BSOD loop and you cannot do the workaround fix if it cannot get in safe mode (Bitlocker).

This is like the ultimate cybersecurity incident. You can bet companies around the world will be reviewing their BCP and disaster recovery plans on Monday.

[Bigtime]

Our POS provider just supplied another update, they figure another 5-6 hours to really get things back to normal. Despite the fixes supplied by Crowdstrike and Microsoft there is a lot of additional troubleshooting and remediation required.

[Fuzz]

I don't think there are any systems that can't be fixed, it is just a pain. Even the bitlocker ones can be done without the bitlocker key by using bcdedit.

1. Cycle through BSODs until you get the recovery screen.
2. Navigate to Troubleshoot > Advanced Options > Startup Settings
3. Press Restart
4. Skip the first Bitlocker recovery key prompt by pressing Esc
5. Skip the second Bitlocker recovery key prompt by selecting Skip This Drive in the bottom right
6. Navigate to Troubleshoot > Advanced Options > Command Prompt
7. Type bcdedit /set {default} safeboot minimal. then press enter.
8. Go back to the WinRE main menu and select Continue.
9. It may cycle 2-3 times.
10. If you booted into safe mode, log in per normal.
11. Open Windows Explorer, navigate to C:\Windows\System32\drivers\Crowdstrike
12. Delete the offending file (STARTS with C-00000291*. sys file extension)
13. Open command prompt (as administrator)
14. Type bcdedit /deletevalue {default} safeboot, then press enter. 5. Restart as normal, confirm normal behavior.

https://old.reddit.com/r/crowdstrike...strike_update/


But questions will be asked(yelled).

[Firebot]

Originally Posted by Fuzz

I don't think there are any systems that can't be fixed, it is just a pain. Even the bitlocker ones can be done without the bitlocker key by using bcdedit.

Yes sure, but do this 5000 times physically on each device at a larger size company on a Friday where most people are working from home and staff are on vacation and have branch offices across the country with outsourced IT

[Fuzz]

Oh, it's absolutely a disaster, I just don't think it's an unsolvable data loss event. I do think some people will be reconsidering where their bitlocker keys get stored, though.

[chemgear]

Originally Posted by activeStick

I'm guessing some genius decided his code change was good and pushed it direct to prod

#### it, hit enter and leave early for the weekend!

[Shazam]

Originally Posted by Azure

https://twitter.com/user/status/1814233715641389456

Absolutely ####ing hilarious.

[woob]

A friend's company got hit by this here in town. 500+ systems. Their team started fixing around 7am and finished up around 6pm. He said it was mind numbing work, entering BL keys for approx 11 hours. Happy to have it all sorted, but I'm sure there's a seething, underlying rage for CrowdStrike within their team.

[Firebot]

Have to feel for this guy.

https://www.reddit.com/r/crowdstrike...t=share_button

[Azure]

Protect yourself from ransomware by installing Crowdstrike!

Except when Crowdstrike decides they have no clue what they're doing and BSODs your entire IT infrastructure just because they can!

Reading the posts on Reddit where IT staff are dealing with this, goodness sake. Crowdstrike should get sued to oblivion and die in a fire.