|
A few days ago, a bad update caused 8.5 million computers to stop working, resulting in $5.4 billion in losses for Fortune 500 companies alone. Some businesses still haven’t recovered, even after nearly two weeks! Chances are you weren’t directly impacted, as the software that caused this mess, Crowdstrike, is more commonly used by larger enterprises. This week, I’m here to emphasize that it’s not a matter of if, but when you and your business will be affected by a major technology outage like this one. While this may seem like a technology issue, it’s actually about business decisions you need to make now to reduce unnecessary risk and exposure. Let’s take this opportunity to learn from this situation so that when crises strike, you are well prepared.
So, What Exactly Happened?
Cybersecurity programs, such as virus protection, have deep-level access to the Windows operating system to respond to threats. Much like your office manager having access to your bank accounts and other sensitive business information to handle daily tasks. When Crowdstrike sent a faulty update to their security software, it caused the Windows operating system to crash. Now imagine your operations manager making a catastrophic decision with all that access to your business. Although the fix wasn’t very complex, the situation quickly worsened because you can’t remotely access a crashed computer to apply the fix. Someone had to be physically present at the computer to enter a computer-specific encryption key (secondary password) and type a couple of commands. However, this was no easy task for many. Some companies couldn’t locate their encryption keys because their IT company hadn’t properly stored them, while others couldn’t get IT support people to show up or guide them through the steps in a for quite some time. At the time I’m writing this, it has been 10 days since this issue started, and Delta Airlines is still canceling hundreds of flights while trying to recover all of their systems.
How Can I Get Ready for Something Like This?
Much like in real life, taking the right steps can help avoid painful situations but can’t eliminate them altogether. The key is being well-prepared for when things go south. If there’s one message to take away from this article, let it be to always have a backup plan. Does each member of your team know who to call during a technology crisis? When you place that emergency call to your IT partner, what should you expect the First Response Time (FRT) and Time to Resolution (TTR) to be? If someone from IT needs to come out, how quickly can your IT partner have boots on the ground? Large IT providers, much like large software companies such as Crowdstrike, often fall short in dispatching a speedy response and providing personal service. This is why, whenever possible, we choose to work with partners who hold our business in the highest regard.
There is, however, one more lesson from the Crowdstrike failure. Even though it wouldn’t have stopped this catastrophe, since Crowdstrike pushes their updates directly, we frequently prevent major computer issues by restricting faulty updates from installing. Updates are extremely important as long as they don’t break some things while fixing others. Whenever possible, make sure updates you choose to install on your systems are tested and known to be free from defects. This is something your IT partner should already be doing for you.
Whether it’s picking the right IT partner or investing in the right team member, making the right choices in your business will always come to your rescue in preventing fires as well as putting them out quickly.
Stay Safe,
– Burak Sarac, Team Lead
|
