[SOLVED] UEFI laptop won't reboot after attempted kernel upgrade (-current)

semiprime · 01-20-2021, 07:21 AM

I've messed up a kernel upgrade on current. Feels like a rite of passage for all Slackware users! Any help would be appreciated.

Background (from memory): I got a new laptop in December, so a recent kernel was required, so I decided to go for Slackware-current rather than 14.2.

I got AlienBOB's current iso, and put it on a USB stick. This was from about December 10th, kernel version 5.4.82. (I still have this stick to boot from).

All installed fine. I wiped the Windows10 partition. Partitions sda1: efi, sda2: swap, sda3: Slackware root. I used the huge kernel and didn't bother making an initrd (put off to later).

Skip forward to January. I've been putting off upgrading until the new kernel 5.10.x seemed stable. I have a snapshot of the Slackware packages from Dec 17 (kernel 5.4.84), so since I'm not used to UEFI, I decide to upgrade to that as a test before jumping to kernel 5.10.x.

I upgrade all using upgradepkg --install-new. I removepkg three packages listed as "removed" in the changelog. I run eliloconfig [select all the defaults]. I don't make a new initrd, since I'm using the huge kernel.

Reboot and ... it fails, complaining about the lack of 5.4.84 modules. Uh-oh. Maybe I needed that initrd.

So I reboot from the install iso to try to repair things, and after a bit of experimentation I do the following.

Chroot:

Code:

mount /dev/sda3 /mnt
chroot /mnt /bin/bash

Mount things:

Code:

mount -t proc none /proc
mount -t sysfs none /sys
mount /boot/efi

Generate an initrd:

Code:

cd /boot
/usr/share/mkinitrd/mkinitrd_command_generator.sh -k 5.4.84 -i
# copy + run the output of mkinitrd_command_generator.sh:
mkinitrd -c -k 5.4.84 -f ext4 -r /dev/sda3 -m usb-storage:xhci-hcd:jbd2:mbcache:crc32c_intel:crc32c_generic:ext4 -l uk -u -o /boot/initrd.gz

No errors, so I run eliloconfig and choose all the defaults.

Checking /boot/efi/EFI/Slackware/ I see that indeed the generic kernel has been copied there, along with the initrd. Although /boot/vmlinuz points to vmlinuz-huge-5.4.84.

Trying to reboot, "shutdown now" doesn't work (no /dev/initctl) "poweroff" doesn't work (can't determine runlevel). So I hold down the power key until it powers down. Remove install usb. Reboot.

I get a Windows10 style screen (blue background, font that looks like Windows) that says:

Quote:

Recovery

Your PC/Device needs to be repaired

A required device isn't connected or can't be accessed.

Error code: 0xc0000225

You'll need to use recovery tools. If you don't have any installation media (like a disc
or USB device), contact your PC administrator or PC/Device manufacturer.

Press Enter to try again
Press F1 to enter Recovery Settings
Press F8 for Start-up Settings
Press Esc for UEFI Firmware Settings

This is where I'm stuck. I hope the hard power-down didn't corrupt the UEFI partition.

keefaz · 01-20-2021, 07:42 AM

You don't see Slackware entry when pressing F8 at boot?

Maybe you need to add manually EFI boot entry, try to boot in installed system on internal HD with your media installer
Then once logged as root:

Code:

modprobe efivars
efibootmgr -c -L Slackware -l '\EFI\Slackware\elilo.efi'
# remove media installer
reboot

semiprime · 01-20-2021, 08:47 AM

Quote:

Originally Posted by keefaz

You don't see Slackware entry when pressing F8 at boot?

F8 at the bluescreen error gives a similar screen, with the additional text:

The application or operating system couldn't be loaded because a required file is
missing or contains errors.

File: \Windows\system32\winload.efi
Error code: 0xc0000225

If I press Esc after switching the laptop on and get a boot menu, selecting the Slackware option takes me to the BIOS Utility - maybe a fallback?

Quote:

Originally Posted by keefaz

Maybe you need to add manually EFI boot entry, try to boot in installed system on internal HD with your media installer
Then once logged as root:

Code:

modprobe efivars
efibootmgr -c -L Slackware -l '\EFI\Slackware\elilo.efi'
# remove media installer
reboot

Trying that...

I find there's no need to modprobe efivars with the install media kernel, so I can just chroot and run efibootmgr. I get the following:

Code:

efibootmgr: ** Warning ** : Boot0000 has same label Slackware
BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0001,0000,0003,0004,0005
Boot0000* Slackware HD([Stuff snipped - mostly hex])/File(\EFI\SLACKWARE\ELILO.EFI)
Boot0003* Windows Boot Manager [details snipped]
Boot0004* UEFI: KingstonDT 101 G2 1.00 [snipped - this is the install USB]
Boot0005* UEFI: KingstonDT 101 G2 1.00 Partition 1 [Snipped]
Boot0001* Slackware HD([Stuff snipped - same as Boot0000 above])/File(\EFI\Slackware\elilo.efi)

Then rebooting gives me the same bluescreen as before.

One thing I noticed. If I mount /boot/efi and do file /boot/efi/EFI/Slackware/* then I get error messages for all the files:

Quote:

ERROR: cannot read '/boot/efi/EFI/Slackware/elilo.conf' (input/output error)

This doesn't happen for the files under /boot/efi/EFI/Microsoft/.

Running eliloconfig seems makes the files readable again (the file command works), but if I umount and mount /boot/efi I get the input/output errors again.

(Thing I learned: how to reboot after chroot - exit then reboot as normal.)

ReFracture · 01-20-2021, 08:49 AM

Ah this old issue...

I'm not saying this is best practice or anything, but when there's a kernel update all I do is copy the vmlinuz-huge-5.x.x from /boot to /boot/efi/EFI/Slackware/ named as just vmlinuz.

My elilo.conf

Code:

chooser=simple
delay=1
timeout=1
#
image=vmlinuz
        label=vmlinuz
#       initrd=initrd.gz
        read-only
        append="root=/dev/sda1 vga=normal ro"

I commented out the initrd since I'm using kernel-huge. root path obviously may differ for others.

Is this best practice? I don't know. Running liloconfig on an MBR system after kernel upgrade works perfectly for me, it always reboots with the upgraded kernel and works great.. never once had to do anything else. Running eliloconfig on an EFI system after kernel upgrade does not work out for me, it always results in a mounting error followed by a kernel panic.

Apart from the need to reinstall nvidia drivers after kernel upgrades this is what largely prevents me from updating my kernel more than once every couple of months.

I'm not sure if any of this is helpful, laptops can be weird with all their partitions the manufacturer try to hide on it for recovery purposes and whatnot. When I got my Dell Inspiron i7559 I actually replaced the spinning drive it shipped with for an SSD, so I haven't had to deal with that.

keefaz · 01-20-2021, 09:09 AM

I/O error suggests hardware failure of some sort
I would boot media installer then verify disk structure

Code:

lsblk -f

Also a filesystem check for sda1

Code:

fsck /dev/sda1

Say no to question if in doubt, maybe ask here

semiprime · 01-20-2021, 09:17 AM

Quote:

Originally Posted by ReFracture

Ah this old issue...

I certainly hope I've learned someting that will make my next kernel update a little smoother! I just have to work out how to get my laptop booting again. Preferably without doing a full reinstall.

semiprime · 01-20-2021, 09:27 AM

Quote:

Originally Posted by keefaz

I would boot media installer then verify disk structure

lsblk doesn't reveal anything unusual.

fsck /dev/sda1 gives:

Code:

fsck from util-linux 2.36.1
CP437: Invalid argument
fsck.fat 4.1 (2017-01-24)
Seek to 272629248: Invalid argument

colorpurple21859 · 01-20-2021, 09:30 AM

Quote:

it fails, complaining about the lack of 5.4.84 modules

What is in /lib/modules?

keefaz · 01-20-2021, 09:43 AM

Quote:

Originally Posted by semiprime

lsblk doesn't reveal anything unusual.

fsck /dev/sda1 gives:

Code:

fsck from util-linux 2.36.1
CP437: Invalid argument
fsck.fat 4.1 (2017-01-24)
Seek to 272629248: Invalid argument

Is there any error output in dmesg command after you mount sda1?

Code:

# on booted media installer system

mkdir /bla
mount /dev/sda1 /bla
dmesg

If sda1 can be mounted, I would be tempted to save any file from it to an external media drive and reformat /dev/sda1 in vfat format, then copy the saved files to it

semiprime · 01-20-2021, 09:47 AM

Quote:

Originally Posted by colorpurple21859

What is in /lib/modules?

After my initial attempt to upgrade the kernel, /lib/modules contained just a 5.4.84 directory, as expected. I think it was their absence in any sort of initrd that was the problem.

Now, after my clumsy hacking around:

Code:

ls /lib/modules
5.4.84/
ls /boot/initrd-tree/lib/modules/
5.4.84/

So that should be fine. I think I've caused a different problem while trying to fix the initial problem.

keefaz · 01-20-2021, 10:15 AM

On another thought, maybe the laptop firmware was reset to a default config when abruptly powered off, like enabling secure boot etc
I would review EFI bios settings just in case (press F2 or delete key while powering up the laptop, don't know how you access the settings in your laptop, oh maybe it's the ESC key?)
But it seems there is an issue with /dev/sda1 partition

semiprime · 01-20-2021, 10:36 AM

Quote:

Originally Posted by keefaz

Is there any error output in dmesg command after you mount sda1?

Code:

$ mount /dev/sda1 /mnt
$ dmseg | grep sda1
 sda: sda1 sda2 sda3
FAT-fs (sda1): Volume was not properly unmounted/ Some data may be corrupt. Please run fsck.

That seems to confirm that me now knowing how to reboot after chroot and hitting the power button has corrupted the uefi partition. So my fault, and hopefully not a hardware problem.

Prompted by this, I thought if I ran eliloconfig, then copied the files from EFI/Slackware (now readable - presumably cached after being written) then deleted EFI/Slackware and copied them back, I might get the filesystem into a consistent state.

I renamed EFI/Slackware to EFI/Slackware-corrupt. unmounted and renounted. I could now read the files in EFI/Slackware-corrupt! So I renamed the directory back to EFI/Slackware, and it remained readable over a umount/mount. fsck still gave errors but it seemed "better".

Cautiously, I did clean powerdown. And the laptop rebooted into kernel 5.4.84.

I have a corrupt filesystem on my uefi partition, but it's better. I'll think about what to do about that - maybe a full reinstall when a 15.0 release candidate arrives and I can then reformat /dev/sda1 to prevent it causing trouble later.

I'll mark this as "Solved" -- thanks keefaz!

My only question now is: why did running eliloconfig try to use the generic kernel?

keefaz · 01-20-2021, 10:52 AM

I do have the same message about volume not properly unmounted on my system although it always has been shutdown properly, it's a bit that can be cleared with fsck, but I never bothered with it

keefaz · 01-20-2021, 10:58 AM

Quote:

Originally Posted by keefaz

I do have the same message about volume not properly unmounted on my system although it always has been shutdown properly, it's a bit that can be cleared with fsck, but I never bothered with it

Reviewing the script, it copies /boot/vmlinuz in /boot/efi/EFI/Slackware and it tests if there is /boot/initrd.gz file, if yes it copies it as well and add the initrd configuration in elilo.conf, so if you want to use eliloconfig with huge, make sure /boot/vmlinuz points to it and delete /boot/initrd.gz

semiprime · 01-20-2021, 11:24 AM

Quote:

Originally Posted by keefaz

Reviewing the script, it copies /boot/vmlinuz in /boot/efi/EFI/Slackware and it tests if there is /boot/initrd.gz file, if yes it copies it as well and add the initrd configuration in elilo.conf, so if you want to use eliloconfig with huge, make sure /boot/vmlinuz points to it and delete /boot/initrd.gz

That's a bit odd - in my first post I observed:

Quote:

Originally Posted by semiprime

Checking /boot/efi/EFI/Slackware/ I see that indeed the generic kernel has been copied there, along with the initrd. Although /boot/vmlinuz points to vmlinuz-huge-5.4.84.

Maybe I was mistaken. The evidence of what happened has been overwritten a few times now.