Software Update | Kami Vaniea

Crowdstrike Outage - thoughts about updates

Fri, 26 Jul 2024 00:00:00 +0000

Shortly after the CrowdStrike incident happened, the Cybersecurity and Privacy Institute at University of Waterloo helped publish an article on “ ” by Kami Vaniea (me) and Regina Ashna Singh.

Below is my more detailed views.

At its core the Crowdstrike outage happened because two related issues were not handled well: testing and update management.

What happened

Normally when software needs to be changed, the vendor creates an update, tests it and release it for other people to install. Large organizations that depend on software running correctly typically conduct further tests on the new update before installing it on their systems. Airlines, for example, have dedicated test machines they will install the update on first to make sure it is working. Even then they will likely also do “staged deployment” which means they will update the least critical computers first, make sure everything is operating as expected, and then update the more critical computers. These tests are done because even assuming the original software vendor did an excellent job of testing, the large organization may have specialized software installed that also needs to be tested against.

Security tools, like Anti-virus, often bypass these tests when they are updating configuration files rather than software code. Anti-virus tools rely on lists of bad things that they are looking for, known as signatures. These lists are sets of patterns or computer-friendly descriptions of what known bad things look like. Some computer viruses can attack very quickly. in 2003 compromised 75,000 computers in 10 minutes. So when a security company learns about new attacks they want to send out an update to the signature lists quickly and they understandably do not want to wait for testing.

Crowdstrike at some time in the past . Their internal testing should have caught the error, but it did not because it only happens if a certain type of configuration file is present, and that type of configuration file was not used when testing the code. Then on Friday July 19th they pushed an update to the configuration file that caused their code to incorrectly try to read something that was not there (out-of-bounds memory read). Windows correctly identified abnormal behavior and shut down to prevent starting in an unsafe way. In other words, Windows experienced a Blue Screen of Death where it shows a blue screen with an error during startup and has to be shut down.

Normally security issues like this are fixed quickly and the public only has to experience a downtime of a few hours. The Crowdstrike issue took much longer to fix because it damaged how Windows starts up. Because CrowdStrike’s code is security software it has to be loaded and run early in the Windows startup process. Once a Windows computer downloaded the problematic configuration file the only way to start it again is to remove or replace the file. That means that a person has to visit each computer individually, start it in something called ``safe mode’’ and remove the problematic file. This process takes time and cannot be fixed remotely or automatically. There were some innovative solutions, such as the person who realized that automate some of the process using a .

Testing and update management

Due to the incident “add additional deployment layers and acceptance checks” and “allow customers to control how Rapid Response Content is deployed”. In other words, they have decided to no longer release configuration files wordwide all at once and instead do a form of staged deployment. They have also decided to allow the system administrators at companies to decide when and how the configuration files are deployed on their own computers. So it may now be possible for admins to test configuration files in advance of deploying them.

Updates to original article

Notably the went to Facebook with Crowdstrike coming in second globally. What is most notable is the breakdown by region. Crowdstrike is number two for US and Canada, but isn’t even in the top ten for other regions.

To Patch, or not To Patch? That is the Question: A Case Study of System Administrators' Online Collaborative Behaviour

Sun, 01 Jan 2023 00:00:00 +0000

Patch Management

Tue, 27 Apr 2021 00:00:00 +0000

Updating software is one of the most effective methods of protecting computers from security vulnerabilities, yet many people and organisations choose to not install them. In this project we endevor to understand the attitudes of people making update decisions and the barriers they face that can impact their willingness and ability to keep their software up to date.

Updating software means changing it. Ideally those changes are for the better such as improved security, better features, fixed bugs, and additional functionality. However, they can also lead to worse outcomes such as slower software, new problems, changes to frequently used functionality, and new security vulnerabilities. According to our research, many of the issues around software updating are actually issues around balancing risks and benefits when neither of these are fully known. For example, the update that protected against the WannaCry ransomeware attack was made available a month before the attack happened, those that installed it were therefore protected and not impacted by the attack. But it would have been impossible for a system administrator to know at the time of update release that a devistating world wide ransomware attack was comming. Incidents like WannaCry suggest that installing patches quickly is a good idea, which is true, but it should also be understood that installing patches quickly also carries risks. For example, Microsoft released a Windows 10 update which then started deleting all the files on the desktop. They acknowledged the issue and offered a work around, but the issue was terrifying and highly disruptive for users, particularly those who installed updates quickly.

"Anyone Else Seeing this Error?": Community, System Administrators, and Patch Information

Mon, 01 Jun 2020 00:00:00 +0000

Out of the loop: How automated software updates cause unintended security consequences

Wed, 01 Jan 2014 00:00:00 +0000