It’s not really criticism, it’s competitors claiming they will never fuck up.
Like, if you found mouse in your hamburger at McDonald’s, that’s a massive fuckup. If Burger King then started saying “you’ll never find anything gross in Burger King food!” that would be both crass opportunism and patently false.
It’s reasonable to criticize CrowdStrike. They fucked up huge. The incident was a fuckup, and creating an environment where one incident could cause total widespread failure was a systemic fuckup. And it’s not even their first fuckup, just the most impactful and public.
But also Microsoft fucked up. And the clients, those who put all of their trust into Microsoft and CrowdStrike without regard to testing, backups, or redundancy, they fucked up, too. Delta shut down, cancelling 4,600 flights. American Airlines cancelled 43 flights, 10 of which would have been cancelled even without the outage.
Like, imagine if some diners at McDonald’s connected their mouths to a chute that delivers pre-chewed food sight-unseen into their gullets, and then got mad when they fell ill from eating a mouse. Don’t do that, not at any restaurant.
All that said, if you fuck up, you don’t get to complain about your competitors being crass opportunists.
It’s not really criticism, it’s competitors claiming they will never fuck up.
Not in all cases [podcast warning], sometimes it’s just them pointing out they’re doing silly things like how they test every update and don’t let it out the door with <98% positive returns or having actual deployment rings instead of of yeeting an update to millions systems in less than an hour.
It’s reasonable to criticize CrowdStrike. They fucked up huge. The incident was a fuckup, and creating an environment where one incident could cause total widespread failure was a systemic fuckup. And it’s not even their first fuckup, just the most impactful and public.
Clownstrike deserves every bit of shit they’re getting, and it amazes me that people are buying the bullshit they’re selling. They had no real testing or quality control in place, because if that update had touched test windows boxes it would have tipped them over and they’d have actually known about it ahead of time. Fucking up is fine, we all do it. But when your core practices are that slap dash, bitching about criticism just brings more attention to how badly your processes are designed.
But also Microsoft fucked up.
How did Microsoft fuck up? Giving a security vender kernel access? Like they’re obligated to from previous lawsuits?
And the clients, those who put all of their trust into Microsoft and CrowdStrike without regard to testing, backups, or redundancy, they fucked up, too
Customers can’t test clownstrike updates ahead of time or in a nonprod environment, because clownstrike knows best lol.
Redundancy is not relevant here because what company is going to use different IDR products for primary and secondary tech stacks?
Backups are also not relevant (mostly) because it’s quicker to remediate the problem than restore from backup (unless you had super regular DR snaps and enough resolution to roll back from before the problem.
IMO, clownstrike is the issue, and customers have only the slightest blame for using clownstrike and for not spending extra money on a second IDR on redundant stacks.
In what way did Microsoft fuck up? They don’t control Crowdstrike updates. Short of the OS files being immutable it seems unlikely they can stop things like this.
Microsoft gave CrowdStrike unfettered access to push an update that can BSOD every Windows machine without a bypass or failsafe in place. That turned out to be a bad idea.
CrowdStrike pushed an errant update. Microsoft allowed a single errant update to cause an unrecoverable boot loop. CrowdStrike is the market leader in their sector and brings in hundreds of millions of dollars every year, but Microsoft is older than the internet and creates hundreds of billions of dollars. CrowdStrike was the primary cause, but Microsoft enabled the meltdown.
Microsoft did not “give Crowdstrike access to push updates”. The IT departments of the companies did.
The security features that Crowdstrike has forces them to run in kernel-space, which means that they will have code running that can crash the OS. They crashed Debian in an almost identical way (forced boot loop) about a month before they did the same to Windows.
Yes, there are ways that Microsoft could rewrite the Windows kernel architecture to make it resistant to this type of failure. But I don’t think there are very many other commercial OS’s that could stop this from happening.
Microsoft gave CrowdStrike unfettered access to push an update that can BSOD every Windows machine without a bypass or failsafe in place. That turned out to be a bad idea.
Microsoft software licensing expert Rich Gibbons said: “Microsoft has received some criticism for the fact that a third party was able to affect Windows at such a deep technical level. It’s interesting that Microsoft has pointed out the fact this stems from a 2009 EU anti-competition ruling that means Microsoft must give other security companies the same access to the Windows kernel as they have themselves.”
Even if that’s the case, how is it Crowdstrike’s place to call these other companies out for claiming something similar will never happen to them? Thus far, it had only ever happened to CS.
No, we had Sentinelone take down our company a few months ago. Granted, not a global outage, but it’s something similar. I’m sure that if you went back in news archives, you’d find articles about major Sentinelone outages. I think Crowdstrike is just the biggest one in recent history. It’s certainly not unprecedented.
It’s not really criticism, it’s competitors claiming they will never fuck up.
Like, if you found mouse in your hamburger at McDonald’s, that’s a massive fuckup. If Burger King then started saying “you’ll never find anything gross in Burger King food!” that would be both crass opportunism and patently false.
It’s reasonable to criticize CrowdStrike. They fucked up huge. The incident was a fuckup, and creating an environment where one incident could cause total widespread failure was a systemic fuckup. And it’s not even their first fuckup, just the most impactful and public.
But also Microsoft fucked up. And the clients, those who put all of their trust into Microsoft and CrowdStrike without regard to testing, backups, or redundancy, they fucked up, too. Delta shut down, cancelling 4,600 flights. American Airlines cancelled 43 flights, 10 of which would have been cancelled even without the outage.
Like, imagine if some diners at McDonald’s connected their mouths to a chute that delivers pre-chewed food sight-unseen into their gullets, and then got mad when they fell ill from eating a mouse. Don’t do that, not at any restaurant.
All that said, if you fuck up, you don’t get to complain about your competitors being crass opportunists.
Not in all cases [podcast warning], sometimes it’s just them pointing out they’re doing silly things like how they test every update and don’t let it out the door with <98% positive returns or having actual deployment rings instead of of yeeting an update to millions systems in less than an hour.
Clownstrike deserves every bit of shit they’re getting, and it amazes me that people are buying the bullshit they’re selling. They had no real testing or quality control in place, because if that update had touched test windows boxes it would have tipped them over and they’d have actually known about it ahead of time. Fucking up is fine, we all do it. But when your core practices are that slap dash, bitching about criticism just brings more attention to how badly your processes are designed.
How did Microsoft fuck up? Giving a security vender kernel access? Like they’re obligated to from previous lawsuits?
Customers can’t test clownstrike updates ahead of time or in a nonprod environment, because clownstrike knows best lol.
Redundancy is not relevant here because what company is going to use different IDR products for primary and secondary tech stacks?
Backups are also not relevant (mostly) because it’s quicker to remediate the problem than restore from backup (unless you had super regular DR snaps and enough resolution to roll back from before the problem.
IMO, clownstrike is the issue, and customers have only the slightest blame for using clownstrike and for not spending extra money on a second IDR on redundant stacks.
In what way did Microsoft fuck up? They don’t control Crowdstrike updates. Short of the OS files being immutable it seems unlikely they can stop things like this.
Microsoft gave CrowdStrike unfettered access to push an update that can BSOD every Windows machine without a bypass or failsafe in place. That turned out to be a bad idea.
CrowdStrike pushed an errant update. Microsoft allowed a single errant update to cause an unrecoverable boot loop. CrowdStrike is the market leader in their sector and brings in hundreds of millions of dollars every year, but Microsoft is older than the internet and creates hundreds of billions of dollars. CrowdStrike was the primary cause, but Microsoft enabled the meltdown.
Microsoft did not “give Crowdstrike access to push updates”. The IT departments of the companies did.
The security features that Crowdstrike has forces them to run in kernel-space, which means that they will have code running that can crash the OS. They crashed Debian in an almost identical way (forced boot loop) about a month before they did the same to Windows.
Yes, there are ways that Microsoft could rewrite the Windows kernel architecture to make it resistant to this type of failure. But I don’t think there are very many other commercial OS’s that could stop this from happening.
You’re absolutely right, here is an in-depth explanation from Dave Plummer, the guy who wrote the task manager: https://youtu.be/ZHrayP-Y71Q
They have to give that access by EU ruling:
Even if that’s the case, how is it Crowdstrike’s place to call these other companies out for claiming something similar will never happen to them? Thus far, it had only ever happened to CS.
No, we had Sentinelone take down our company a few months ago. Granted, not a global outage, but it’s something similar. I’m sure that if you went back in news archives, you’d find articles about major Sentinelone outages. I think Crowdstrike is just the biggest one in recent history. It’s certainly not unprecedented.