Fix initramfs from broken update
Overview
Update: Did this with Almalinux 8 which is RHEL8 / Rocky Linux 8 equivalent and the below instructions apply to those as well. This was done with hardware server through IPMI.
You can ran into this problem with failed updates and also by editing your mdadm.conf to rule out your root md device, which i managed to do in my last endevour. Be sure to update your mdadm.conf to exclude any DEVICE or other statements that would exclude your root device.
I updated my CentOS 7 virtual guest and after boot it could not locate my partitions. I had actually 2 instances, an LVM guest and a non-LVM guest. LVM guest waited for a few minutes until it told me some errors about dracut. Non-LVM guest waited indefinitely for vda partition mount.
In practice you may see this kind of error message:
dracut-initqueue timeout
And in detail i just got this on my screen :)
dracut-initqueue[247]: Warning: dracut-initqueue timeout - starting timeout scripts
and get thrown into dracut shell.
Solution
I booted the virtual instance with CentOS 7 / Almalinux 8 ISO, selected rescue mode, let it mount existing system as read-write to /mnt/sysimage and chrooted to it.
While chrooted I rebuilt my initramfs with following command:
dracut -H -f /boot/initramfs-3.10.0-1127.8.2.el7.x86_64.img 3.10.0-1127.8.2.el7.x86_64 || dracut -fv --regenerate-all
Where 3.10.0-1127.8.2.el7.x86_64 is the kernel version, which you must check before issuing the command. Using uname is not recommended as you are doing this with a rescue kernel.
After that just sync and reboot. You may need to reboot twice.
Mounting root partition
If your rescue disk fails to mount your OS partitions here is some advice on how to do it manually.
Firstly locate the devices the OS devices are in. Try command “lsblk”, “lvs” and “lvdisplay” to locate the devices. You can locate lvs volumes under /dev/volumegroupname/volumename.
Then mount the hardware devices under /mnt/sysimage. Some examples below:
mount /dev/xxx /mnt/sysimage mount /dev/xxx /mnt/sysimage/var mount /dev/xxx /mnt/sysimage/boot
Then mount the proc, sys, dev filesystems from parent OS:
mount -t proc none /mnt/sysimage/proc mount --rbind /sys /mnt/sysimage/sys mount --rbind /dev /mnt/sysimage/dev mount --rbind /run /mnt/sysimage/run mount --rbind /sys /mnt/sysimage/sys
Comments
All comments and corrections are welcome.