It’s a big one

In a global outage that is about as big as they come CrowdStrike made an update that has incapacitated Windows systems around the world. Individually the fix is not so taxing but in an enterprise with 1000s of endpoints down and a handful of IT workers to fix them it’s a mammoth task.

The Fix

  • Boot Windows into Safe Mode or the Windows Recovery Environment.
    • Restart and press F8 repeatedly (May be F4 or F5).
  • Navigate to the C:\Windows\System32\drivers\CrowdStrike directory.
    • Similar to normal Windows file exploring.
  • Locate the file matching “C-00000291*.sys” and delete it.
    • Make sure you find the right one.
  • Reboot the host normally.

Why that file?

Deleting that specific CrowdStrike driver file likely fixes the BSOD because:

  • The file may be corrupted or incompatible with the current system configuration.
  • It could be conflicting with other drivers or system components.
  • Removing it allows Windows to use a default or fallback driver instead.
  • The BSOD was potentially caused by an issue within that particular CrowdStrike driver file.

Solution for Automating This?

I came across a post on the r/CrowdStrike thread for this problem. It claims to have an automated solution to this problem for enterprise environments.

  • Create a modified WinPE image
  • Add command to startnet.cmd in WinPE image:
    • del C:\Windows\System32\drivers\CrowdStrike\C-00000291*.sys
  • Exit.
  • Set up PXE server with modified WinPE image.
  • Configure affected systems to boot from network.
  • Systems boot from PXE server.
  • WinPE environment loads on target systems.
  • startnet.cmd executes, deleting problematic driver.
  • Systems automatically reboot.
  • Normal boot process resumes without CrowdStrike issue.

WinPE

A modified WinPE (Windows Preinstallation Environment) image is a customized version of Microsoft’s lightweight operating system used for deployment, recovery, and troubleshooting. It’s tailored to include specific drivers, tools, or scripts to meet particular needs. Modified WinPE images are often used by IT professionals for tasks like system deployment or data recovery.

PXE Server

A PXE (Preboot Execution Environment) server allows network-based booting and installation of operating systems on client computers. It provides boot images and configuration files over the network, enabling diskless workstations or computers without local boot media to start up and install an OS remotely. PXE servers are commonly used in large-scale deployments and network management.

Intel i350 T4
Using Storage Sense