Hanging Bootloaders

Last week, security company Eclypsium released details of the "BootHole" vulnerability in Grub2, the almost ubiquitous boot loader of choice on Linux systems. The vulnerability allows attackers to execute arbitrary code during the boot process, potentially establishing long-term persistent presence with pre-OS privileges. Details were released responsibly and timed with the release of patches.

Then those patches bricked a whole lot of machines.  

Starting with the Redhat and CentOS communities, but followed shortly afterwards by those of every major Linux distribution, came reports of computers frozen on blank screens and rendered unbootable (there was actually some speculation this might occur with automated updates, given the complexities of managing UEFI secure boot).

The recommendation from Linux vendors thus far has been to revert the patches (see below).  This has created a sticky situation where technical details of a vulnerability have been widely published but the fix is not yet widely available.

When Bootloaders Go Bad

Although it is possible to manually address the underlying issues, most large organizations will realistically depend on reliable automation and support from their Linux vendors.  Until those issues can be resolved, there are at least two effective forms of mitigation.

• Use file integrity monitoring tools to ensure that bootloader / EFI partitions have not been tampered with
• Root access is a prerequisite for exploiting BootHole, so keep systems in check by updating and securing all other parts of the operating system and applications

Booting Back

Anyone left with an unbootable system as a result of the bootloader patches can follow the steps below to restore a system.  NOTE: this was tested on CentOS 7, other versions of Fedora/Redhat/CentOS may vary slightly. Other distributions will differ.

Boot to Rescue Mode

Using a live/rescue image of the latest version of the effected OS on bootable media (e.g. DVD/USB), boot the system into rescue mode.  If rescue mode is successful, the system image should be mounted under /mnt/sysimage.  If this is not the case, it may be necessary to manually mount the required filesystems, such as /boot, /boot/efi, and the root filesystem, into the appropriate locations.  

Note that LUKS encrypted filesystems requiring key-files to unlock can sometimes be frustrating to work with and troubleshoot.  Unlock passphrases can be temporarily added to LUKS key slots temporarily if this is an issue.

Chroot  

Change the apparent root directory to the mounted system image.

    chroot /mnt/sysimage

Configure Networking

In rescue mode networking will usually need to be configured manually.

    ifconfig <adapter> <IP address> up
    route add deafult gw <gateway IP address>

It may also be necessary to edit /etc/resolv.conf and add an entry with a DNS resolver

    nameserver <DNS server>

Downgrade Packages

The following commands will re-install working versions of the bootloader.

    yum downgrade shim-x64 mokutil grub2\* #downgrade packages
    grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg #rebuild grub2 config file (just in case)
    touch /.autorelabel #force necessary relabeling of SELinux contexts on the next reboot

Wrap it Up

To temporarily prevent bootloader packages from automatically updating once again, add the below line to /etc/yum.conf (this should be removed once patches are ready and installation procedures have been fully tested).

    exclude=grub2* shim* mokutil

Exit the chroot environment, exit the rescue shell, and reboot.

Update: Redhat and CentOS have now released updated shim packages.  It should therefor be possible to replace yum downgrade with yum update in the recovery steps above (untested by this author).