Ubuntu Server boots to grub rescue after every kernel upgrade, have to revert to older kernel to boot, wrong disk in grub?

surfrock66 · 10-13-2020, 11:03 AM

I have a relatively new Ubuntu server but every time I reboot, I'm sent to GRUB rescue and have to do some weird stuff to get it to work. First, here's my partition layout from within the working server:

Code:

NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 97.7M  1 loop /snap/core/10126
loop1    7:1    0 97.1M  1 loop /snap/core/9993
loop2    7:2    0   55M  1 loop /snap/core18/1880
loop3    7:3    0 68.6M  1 loop /snap/lxd/17597
loop4    7:4    0 55.3M  1 loop /snap/core18/1885
loop5    7:5    0 68.6M  1 loop /snap/lxd/17629
sda      8:0    0  3.7T  0 disk 
├─sda1   8:1    0    1G  0 part 
└─sda2   8:2    0  3.7T  0 part /
sr0     11:0    1 1024M  0 rom

Here's the end of fdisk -l:

Code:

Device       Start        End    Sectors  Size Type
/dev/sda1     2048    2099199    2097152    1G BIOS boot
/dev/sda2  2099200 7809791999 7807692800  3.7T Linux filesystem

When I reboot, it appears that the system is looking for /boot on /dev/sda1, but /boot is actually in /dev/sda2 (which in GRUB is (hdo,gpt2)).

If I look in /boot, I see the following:

Code:

config-5.4.0-45-generic
config-5.4.0-47-generic
config-5.4.0-48-generic
grub
grub.bak
initrd.img
initrd.img-5.4.0-45-generic
initrd.img-5.4.0-47-generic
initrd.img-5.4.0-48-generic
initrd.img.old
memtest86+.bin
memtest86+.elf
memtest86+_multiboot.bin
System.map-5.4.0-45-generic
System.map-5.4.0-47-generic
System.map-5.4.0-48-generic
vmlinuz
vmlinuz-5.4.0-45-generic
vmlinuz-5.4.0-47-generic
vmlinuz-5.4.0-48-generic
vmlinuz.old

So, it reboots, I get brought to GRUB rescue. If I do the following:

Code:

set prefix=(hd0,gpt2)/boot/grub
set root=(hd0,gpt2)
insmod normal
normal
boot

I get:

Code:

Error: you need to load kernel first

If I try to load the kernel with the following:

Code:

insmod linux
linux /boot/vmlinuz root=/dev/sda2

I get:

Code:

error: attempt to read or write outside of disk “hd0″

Instead, if I do the following:

Code:

linux /boot/vmlinuz-5.4.0-45-generic root=/dev/sda2
initrd /boot/initrd.img-5.4.0-45-generic
boot

I'm good to go.

I'm in over my head a bit here, what's the best course of action for getting this so the system can safely reboot after a kernel update? Try to move /boot back to sda1? Change some configuration so it remembers to point to sda2?

syg00 · 10-13-2020, 07:12 PM

Since it's a "relatively new Ubuntu server" it should be EFI rather than BIOS - so you'd need a EFI partition, not a BIOS_boot. Go here and do as it says - that way we'll all have some relevant info.

colorpurple21859 · 10-13-2020, 08:16 PM

Quote:

Try to move /boot back to sda1?

was sda1 mounted to /boot originally, and if so why did you change it? What did you do to change /boot/ to sda2?
Was sda1 an efi parition formatted fat32 originally?
Your probably getting the grub-rescue because you didn't run

Code:

sudo grub-install --force /dev/sda

after moving /boot to /sda2. If that will even work. Not all distros grub-install will install bootloader code to the mbr of gpt type disk. At the grubrescue prompt what is the output of

Code:

set

looking for:
cmdpath=
grub_platform=
prefix=
root=

What is in your /etc/fstab currently and do you know what was in it before you decide to make changes?

surfrock66 · 10-13-2020, 10:47 PM

Here is the output of that script.

colorpurple21859 · 10-14-2020, 05:02 AM

open a terminal

Code:

sudo grub-install /dev/sda

surfrock66 · 10-14-2020, 10:08 AM

Interestingly, this server was hit by an installer bug I submitted:

https://bugs.launchpad.net/subiquity/+bug/1857789

surfrock66 · 10-21-2020, 03:27 PM

After trying to install grub, the server is completely unbootable. If I boot from a live environment, I can chroot into my drive and all my data is there, but the system is totally unbootable.

I tried using boot-repair, but that also did not fix it. It did give me useful debug output though, so I'm not sure if this can help me figure out what to do next:

https://paste.ubuntu.com/p/M5jTK6yVZX/

colorpurple21859 · 10-21-2020, 06:08 PM

give more details, grub-rescue>, grub> prompt, grub-menu but hangs when selecting menu item, something else? What happens when you go into the bios boot menu do you have more than one choice, does the bios boot menu have the option to boot from file?

colorpurple21859 · 10-21-2020, 06:47 PM

post the output of

Code:

parted -l

colorpurple21859 · 10-22-2020, 09:26 AM

I believe for some reason or another the system isn't wanting to boot sda in efi mode. I think this will work for a quick fix.
boot a live iso, With gparted shrink the fat32 partition so you will have a 2mb space between the efi partition and root partition. In the 2Mb space create a new cleard partition labeled bios-boot and flag it as bios_grub,

Code:

mount /dev/sda2 /mnt
grub-install --target=i386-pc --boot-directory=/mnt/boot /dev/sda

surfrock66 · 10-22-2020, 10:36 AM

I made that efi partition 1GB since I had the space, is that too big? Is there a target I should go for; I often see "100-200mb" recommended but haven't found an upper limit.

I'll do those things now, the system is powered off in the rack room.

colorpurple21859 · 10-22-2020, 11:11 AM

Quote:

I made that efi partition 1GB since I had the space, is that too big?

Yes usually it is smaller, the size of the efi partition isn't the issue.

This is just a guess of what is happening, the system isn't allowing sda to boot in efi mode. To allow to boot in legacy mode, You need a 1-2 mb partition cleared/empty with the bios-grub flag set. This will allow grub to install in legacy mode to sda mbr.

The boot code needed for grub to boot in legacy mode on a gpt disk in split between the mbr and the bios-boot partition.
The grub code that goes in bios-boot was located on the efi partition, when "grub-install --target=x86_64-efi" was ran the grub code on the efi partition was overwritten, henceforth unable to boot sda.

surfrock66 · 10-22-2020, 12:33 PM

Ok. I went through the RAID configuration; as far as I can tell everything is correct. It's a 5-disk raid5 made from 1TB disks, which specifically should be ok as the PERC 6/i has an issue with disks over 2TB in size (not virtual disks, individual disks, reference: https://community.spiceworks.com/top...-dell-perc-6-i ). The bootable VD is correct (the only one) and "Enable controller BIOS" is checked. Pictures attached.

I created the partition (it's 16MB because if I didn't do that, the efi partition got moved to start at 14MB on the disk). Picture attached.

I did the grub install exactly as indicated and it said no errors to report. I rebooted, it returned to grub rescue. When I do "ls" I get the following:

(proc) (hd0) (cd0)

When I do set I see "prefix=(hd0)/EFI/ubuntu" but when I do ls of (hd0) or any sub directory I get "error: unknown filesystem."

I guess grub is no longer seeing the gpt partitions? In gparted from the live environment, everything looks fine and I can see all my data. Doing "insmod part_gpt" doesn't help

colorpurple21859 · 10-22-2020, 03:52 PM

Quote:

Ok. I went through the RAID configuration;

I don't see anywhere you mentioned anything about raid until now.

mount /dev/sda2 /mnt
grub-install --target=i386-pc --modules="raid5rec mdraid09 mdraid1x" --boot-directory=/mnt/boot /dev/sda

add this to your /etc/default/grub
GRUB_PRELOAD_MODULES="mdraid09 mdraid1x"

surfrock66 · 10-22-2020, 04:45 PM

Does it matter that it's not software RAID, but a HW RAID through the dell PERC6/i controller? I don't know why I didn't mention it, I think I did in a post somewhere else and just forgot that part here, my apologies.