Lessons from the CrowdStrike Incident – A Commentary

Together with my co-authors Vanessa Chamera and Martin Bodenstein, I have written a commentary on the CrowdStrike incident for Springer Gabler. The incident demonstrated globally how vulnerable IT systems can be, despite extensive security measures, and why backups are often the last line of defense. 

Backups as the Last Bastion

On July 19, 2024, businesses worldwide were brought to a standstill due to a faulty update from the software company CrowdStrike.

Although companies today implement comprehensive IT security measures, even the best firewalls and antivirus programs, along with organizational processes in the background, are not infallible. Human error and technical failures can never be completely ruled out. In such cases, a well-planned backup strategy can be the last line of defense.

The incident on July 19, 2024, when a faulty update from CrowdStrike caused widespread outages, painfully reminded us of the importance of a solid backup strategy. Companies with reliable backups were able to restore their systems to the state before the update and quickly resume operations. However, many companies still underestimate the importance of regular backups and effective disaster recovery plans.

A Security Update with Catastrophic Consequences

CrowdStrike is a leading software company that provides IT security solutions across industries, aiming to minimize the risk of outages and protect IT systems from threats. However, a faulty update to their protection software "CrowdStrike Falcon"[1] led to millions of computers worldwide being incapacitated. The update caused a “Blue Screen of Death” (BSOD) on numerous Windows systems, resulting in temporary shutdowns for many businesses, including critical infrastructures such as airports and hospitals.

A faulty file, released in the early morning hours of July 19, 2024, and deployed into IT systems, was responsible for the chaos. Affected systems had to be manually reset to resume operations. Companies without current backups were forced to resort to time-consuming workarounds, significantly delaying the restoration of normal operations. The financial and reputational damage was immense.

Similar Incidents from the Past

The CrowdStrike incident is not the first of its kind where IT systems were massively affected by software issues or cyberattacks. A look at similar incidents underscores the importance of being prepared for such eventualities:

• Microsoft Exchange Outage (March 2021): A faulty patch[2] led to massive outages in email systems worldwide. Companies that were well-prepared and had up-to-date backups were able to restore their systems quickly and minimize downtime. This incident again highlights how critical backup and recovery plans are to maintain operations.
  
• SolarWinds Hack (December 2020): A cyberattack delivered via a SolarWinds software update led to widespread security incidents. Many companies and government agencies were affected, showing that even trusted software providers can pose risks. Once again, backups were often the last line of defense to restore systems and prevent further damage.
  
• GitLab Data Loss (2017): GitLab suffered a massive data loss, exacerbated by a failed backup. This event illustrates that even professional providers can make mistakes, underscoring the importance of carefully planned and regularly tested backup strategies.
  

Backups as an Essential Security Measure

The CrowdStrike incident clearly shows that regular updates and professional patch management are important, but not enough. Even the best measures can fail, and in such cases, backups are the last line of defense. A comprehensive disaster recovery plan that includes regular and tested backups is essential to quickly restore operations after an incident.

Disaster Recovery Plans: Quick Response in Crisis Situations

Disaster recovery plans are crucial to minimize the impact of an IT outage. These plans describe the processes and responsibilities for restoring critical systems and data. A key component of these plans is regular simulated crisis exercises to ensure that all stakeholders know how to act in an emergency.

A well-maintained backup is the simplest way to recover lost data or restore broken systems. Depending on a company's risk assessment and resources, a backup strategy can range from simple data backups to redundant IT infrastructures that can be quickly activated in the event of an emergency. Companies should choose between "cold," "warm," and "hot" backup environments according to the criticality and availability requirements.

A cold environment covers the basic infrastructure to resume operations, but recovery time may be longer. A warm environment includes pre-installed systems, while a hot environment offers nearly everything needed to quickly resume business operations. When deciding which environment makes sense, risks and costs must be weighed.

Prevention through the Right Backup Strategy

In today's dynamic cyber world, it is crucial that companies regularly review and adjust their backup strategies. A thoughtful backup strategy begins with identifying critical data and selecting appropriate storage media. It is equally important to align the backup cycle with system changes and ensure that backups themselves are protected by IT security measures. Regular tests of backup routines are essential to ensure they function smoothly in an emergency.

The CrowdStrike incident shows that even with carefully planned updates, unforeseen problems can arise. Therefore, every backup strategy should be designed to allow a quick rollback to a functional previous version in case an update fails.

Lessons Learned and Recommendations

In Germany and Europe, this incident has fueled the discussion on cybersecurity and corporate responsibility, especially regarding new EU regulations like the NIS2 Directive, which will significantly tighten IT security requirements. Companies must now ensure that their systems are robust and that security measures are regularly reviewed and adjusted.

CrowdStrike has learned from the incident and announced that future updates will be rolled out in phases to improve error control and minimize the scope of such outages. The German Federal Office for Information Security (BSI) has also called on software providers to ensure that systems can start in a safe mode in the event of critical errors.

The following recommendations can be derived for companies:

• Test Environment: Updates should be tested in a protected, isolated offline environment ("sandbox") to detect potential issues early.
  
• Avoid Automated Updates: Instead of allowing automated updates, they should be manually reviewed and applied under controlled conditions.
  
• Rollback Strategies: It is important to test prepared rollback strategies and quickly implement them in case of problems.
  
• Staggered Updates: Updates should be staggered across different systems to minimize the risk of widespread outages.
  

In conclusion, backups encompass far more than just data storage. They are an essential part of a comprehensive IT security concept that includes disaster recovery plans, IT security measures following recognized standards, and, in some cases, even redundant IT infrastructures.

Sources and Further Reading:

[1] CrowdStrike Falcon is an Endpoint Detection and Response (EDR) software used to defend against cyber threats on endpoints (PCs, laptops, tablets, smartphones, servers). This EDR software monitors and analyzes endpoint behavior to detect potentially suspicious activities. In the event of anomalies, automated responses such as isolating the affected device can be triggered.

[2] A "patch" is a software update that fixes bugs or closes security gaps, making the software more secure and stable.

More details on prevention strategies, disaster recovery plans, data backup strategies, work aids, and templates can be found in our book: 

Data Backup as Part of IT and Cybersecurity

➛

Read More

About the Co-Authors

Vanessa Chamera completed her Master's in Economic Policy Consulting (M.Sc.) at Ruhr University Bochum and gained professional experience in IT security. In addition to her work in Digital Forensics, she specialized in analyzing information security measures for risk prevention.
➡️LinkedIn

Martin Bodenstein holds a degree in Computer Science (Dipl.-Inf.) and an MBA, with professional experience in IT service, security, and project management. With forensic expertise as a foundation, the core of his knowledge lies in the continuous development and professional hardening of information security measures.
➡️LinkedIn