This is an unpopular opinion, and I get why – people crave a scapegoat. CrowdStrike undeniably pushed a faulty update demanding a low-level fix (booting into recovery). However, this incident lays bare the fragility of corporate IT, particularly for companies entrusted with vast amounts of sensitive personal information.
Robust disaster recovery plans, including automated processes to remotely reboot and remediate thousands of machines, aren’t revolutionary. They’re basic hygiene, especially when considering the potential consequences of a breach. Yet, this incident highlights a systemic failure across many organizations. While CrowdStrike erred, the real culprit is a culture of shortcuts and misplaced priorities within corporate IT.
Too often, companies throw millions at vendor contracts, lured by flashy promises and neglecting the due diligence necessary to ensure those solutions truly fit their needs. This is exacerbated by a corporate culture where CEOs, vice presidents, and managers are often more easily swayed by vendor kickbacks, gifts, and lavish trips than by investing in innovative ideas with measurable outcomes.
This misguided approach not only results in bloated IT budgets but also leaves companies vulnerable to precisely the kind of disruptions caused by the CrowdStrike incident. When decision-makers prioritize personal gain over the long-term health and security of their IT infrastructure, it’s ultimately the customers and their data that suffer.
…you don’t have OOBM on every single networked device and terminal? Have you never heard of the buddy system?
You should probably start writing up an RFP. I’d suggest you also consider doubling up on the company issued phones per user.
If they already have an ATT phone, get them a Verizon one as well, or vice versa.
At my company we’re already way past that. We’re actually starting to import workers to provide human OOBM.
You don’t answer my call? I’ll just text the migrant worker we chained to your leg to flick your ear until you pick up.
Maybe that sounds extreme, but guess who’s company wasn’t impacted by the Crowdstrike outage.
I mean, with the exception of the shackles, this is just logistics 101. The more something needs to stay working or not accidentally trigger a huge problem, the more resources you dedicate to picking up where the regular guy left off because the “fleffingbridge transport 1” company’s bus broke down in front of the regular guy and his bus got hit by a train. Solution? New bus, plant some trees. Prevention? Bridges and tunnels aren’t cheap, but clearly we need one there now. We can’t predict the future but we have to do our best to try or - simulated or real - the cost will be paid in blood. Obviously there’s moral limits, but hiring more staff is not in and of itself immoral nor the wrong approach.
If I was in charge of a real life logistics operation, I’d be devastated if anyone died because of me. I can’t say, however, that it can be avoided. Sometimes people die at random, that’s not yet 100% avoidable and might never be, but I do care. I’d hope people who actually end up in logistics could learn to indulge their empathy enough to remember there are lives on the line, but I can’t blame someone for being bitter that the actual work output is purely being fleeced for profit.