[SOLVED] A start job is running for /dev/disk/by-uuid

goumba · 12-08-2023, 05:52 PM

Quote:

Originally Posted by mrmazda

If this stems before fstab plays a role, UUIDs can be no panacea. Did you read that bug? There it's an early appearance device enumeration issue, before kernel loading, a difference between BIOS and Grub,

If systemd is doing its thing, the kernel is loaded. Kernel mounts rootfs (initrd in this case) and starts the init from there.

Quote:

before fstab is read. It doesn't appear to me that fstab is even in a TW initrd:

My apologies as I was tired and apparently truncated my own post.

The "root=" parameter, is parsed by https://www.freedesktop.org/software...enerator.html# to generate the target for mounting root. If the OP has this specified as a UUID, that will be used. That's why we see the dev-by-uuid target fail and not dev-sdxx.

Quote:

Of course, this presumes the journal is persistent (/var/log/journal exists on the target); and, that boot proceeded far enough for mounting /, that it was mounted

Unfortunately it's failing to mount his root. He'd only be able to do this from a recovery console. I've had root filesystems fail to mount and still got a console - perhaps there are special cases with no console? I truly have no idea. Maybe I am just mis-remembering.

Regardless of all that however, the cause here may be similar to that bug, just not the exact issue. The bug mentions specific drivers were moved from built in to module configurations. Maybe a driver they need is being loaded too late or not at all.

Whenever I plug in a USB drive, I get output like this:

Code:

[196201.436311] usb 3-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[196201.436312] usb 3-8: Product: One Touch Hub
[196201.436313] usb 3-8: Manufacturer: Seagate
[196201.436314] usb 3-8: SerialNumber: 
[196201.438182] hub 3-8:1.0: USB hub found
[196201.438445] hub 3-8:1.0: 4 ports detected
[196202.766527] usb 3-8.1: new high-speed USB device number 15 using xhci_hcd
[196202.941927] usb 3-8.1: New USB device found, idVendor=0bc2, idProduct=ab80, bcdDevice= 1.00
[196202.941932] usb 3-8.1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[196202.941933] usb 3-8.1: Product: One Touch Hub
[196202.941935] usb 3-8.1: Manufacturer: Seagate
[196202.941936] usb 3-8.1: SerialNumber: 
[196202.945732] scsi host6: uas
[196202.946603] scsi 6:0:0:0: Direct-Access     Seagate  One Touch Hub    0066 PQ: 0 ANSI: 6
[196202.949699] sd 6:0:0:0: Attached scsi generic sg3 type 0
[196204.430170] sd 6:0:0:0: [sdc] Spinning up disk...

Similar messages and order at boot.

Without a usb storage driver loading (uas in this example), the drive does not get enumerated, and the filesystems not accessible. I see no such lines in OP's output.

FWIW as long as I can recall these have been modules, but perhaps other config changes affect how and if they are loaded (order?).

colorpurple21859-385635 suggested rebuilding a new initrd. It's not impossible the initrd built at upgrade time is missing something (I've had that happen, although rarely). It may not work without further invention (like in that bug report, explicitly specifying the appropriate modules to load).

OP, if you try booting the previous kernel (TW should keep an old kernel around if the config is untouched), does the system boot? You mention trying Ubuntu, but no mention of trying the previous kernel from what I can see. While it doesn't fix the problem, it will at least allow you to use the system. When riding the edge it's sometimes necessary

lattimro · 12-08-2023, 08:05 PM

Quote:

Originally Posted by goumba

If systemd is doing its thing, the kernel is loaded. Kernel mounts rootfs (initrd in this case) and starts the init from there.

My apologies as I was tired and apparently truncated my own post.

The "root=" parameter, is parsed by https://www.freedesktop.org/software...enerator.html# to generate the target for mounting root. If the OP has this specified as a UUID, that will be used. That's why we see the dev-by-uuid target fail and not dev-sdxx.

Unfortunately it's failing to mount his root. He'd only be able to do this from a recovery console. I've had root filesystems fail to mount and still got a console - perhaps there are special cases with no console? I truly have no idea. Maybe I am just mis-remembering.

Regardless of all that however, the cause here may be similar to that bug, just not the exact issue. The bug mentions specific drivers were moved from built in to module configurations. Maybe a driver they need is being loaded too late or not at all.

Whenever I plug in a USB drive, I get output like this:

Code:

[196201.436311] usb 3-8: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[196201.436312] usb 3-8: Product: One Touch Hub
[196201.436313] usb 3-8: Manufacturer: Seagate
[196201.436314] usb 3-8: SerialNumber: 
[196201.438182] hub 3-8:1.0: USB hub found
[196201.438445] hub 3-8:1.0: 4 ports detected
[196202.766527] usb 3-8.1: new high-speed USB device number 15 using xhci_hcd
[196202.941927] usb 3-8.1: New USB device found, idVendor=0bc2, idProduct=ab80, bcdDevice= 1.00
[196202.941932] usb 3-8.1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[196202.941933] usb 3-8.1: Product: One Touch Hub
[196202.941935] usb 3-8.1: Manufacturer: Seagate
[196202.941936] usb 3-8.1: SerialNumber: 
[196202.945732] scsi host6: uas
[196202.946603] scsi 6:0:0:0: Direct-Access     Seagate  One Touch Hub    0066 PQ: 0 ANSI: 6
[196202.949699] sd 6:0:0:0: Attached scsi generic sg3 type 0
[196204.430170] sd 6:0:0:0: [sdc] Spinning up disk...

Similar messages and order at boot.

Without a usb storage driver loading (uas in this example), the drive does not get enumerated, and the filesystems not accessible. I see no such lines in OP's output.

FWIW as long as I can recall these have been modules, but perhaps other config changes affect how and if they are loaded (order?).

colorpurple21859-385635 suggested rebuilding a new initrd. It's not impossible the initrd built at upgrade time is missing something (I've had that happen, although rarely). It may not work without further invention (like in that bug report, explicitly specifying the appropriate modules to load).

OP, if you try booting the previous kernel (TW should keep an old kernel around if the config is untouched), does the system boot? You mention trying Ubuntu, but no mention of trying the previous kernel from what I can see. While it doesn't fix the problem, it will at least allow you to use the system. When riding the edge it's sometimes necessary

I did not post dmesg so I do not understand:

Quote:

I see no such lines in OP's output.

Yes, I tried to boot the previous kernel (6.5.9-1). I did that first thing when I encountered the "endless" boot. Also, I tried "recover" options for the current kernel and the previous kernel. None worked.

mrmazda · 12-08-2023, 08:31 PM

Maybe at this point to ease up on the aggravation: uninstall all 6.6.x kernels, which will help ensure you don't accidentally lose 6.5.9 before you get a 6.6.x that works. sudo zypper al kernel-default-6.5.9* (al is short for addlock) will have a similar effect, except it will continue until you unlock it. I would also lock the 6.5.9 initrd using chattr +i, or at least make an easy-to-get-to backup of the working one, to make sure dracut can't kill your booting entirely with a broken rebuild.

When a new 6.6.x comes along, perhaps it will simply work. This is hardly the only thread about non-booting 6.6 kernels. openSUSE Bugzilla, openSUSE mailing lists and and openSUSE forums each have some, as do Fedora's mailing lists. 6.6 seems to be an unusually adept creator of boot failures among those who observed no trouble with previous series 6.5.x.

I haven't had any trouble with 6.6.x yet, but I've only installed about 10 of them, with more than 40 to go on my TWs and Fedoras.

lattimro · 12-09-2023, 11:25 AM

Quote:

Originally Posted by mrmazda

Maybe at this point to ease up on the aggravation: uninstall all 6.6.x kernels, which will help ensure you don't accidentally lose 6.5.9 before you get a 6.6.x that works. sudo zypper al kernel-default-6.5.9* (al is short for addlock) will have a similar effect, except it will continue until you unlock it. I would also lock the 6.5.9 initrd using chattr +i, or at least make an easy-to-get-to backup of the working one, to make sure dracut can't kill your booting entirely with a broken rebuild.

When a new 6.6.x comes along, perhaps it will simply work. This is hardly the only thread about non-booting 6.6 kernels. openSUSE Bugzilla, openSUSE mailing lists and and openSUSE forums each have some, as do Fedora's mailing lists. 6.6 seems to be an unusually adept creator of boot failures among those who observed no trouble with previous series 6.5.x.

I haven't had any trouble with 6.6.x yet, but I've only installed about 10 of them, with more than 40 to go on my TWs and Fedoras.

I understand you think there would be an issue with the newly installed kernel 6.6.3-1 (corrected from 6.3.1, my apologize)?

I had some concerns in Post #28 any of you may answer please.

goumba · 12-09-2023, 04:02 PM

Quote:

Originally Posted by lattimro

wouldn't enter in a panic state if something will be wrong with initrd? I believe at this point, for some reasons it can't find the root partition UUID=287blabla. Could be a fonts mistmach issue somewhere even when everything seems all right (grub.cfg)? As long as my outputs regarded Geckolinux posted here are from chroot environment how can one be sure what is in the native (Geckolinux) installation?

Fonts have nothing to do with it. You'd just get a warning message and possibly a display that doesn't look great.

Quote:

Originally Posted by lattimro

I did not post dmesg so I do not understand

You did, but you just don't realize it

Those startup messages in your photo are the same dmesg would display.

Quote:

Originally Posted by mrmazda

which will help ensure you don't accidentally lose 6.5.9 before you get a 6.6.x that works.

The odd thing here is that the problem occurs across kernel versions. OP said 6.5.9 no longer works. That shouldn't have been touched.

I also revisited that bug report. The "breaking" change occurred between 6.5.3 and 6.5.4.

Also, as far as the EFI boot partition goes, while it's not recommended, as long as OP consistently mounts the EFI partition and hides (does not mount) the other, they should be alright. OP is using the EFI boot manager to choose which partition to boot from. I've installed distros on flash drives and the installer typically ignores the existing EFI System Partition, so I end up with one internal, one on the drive. Usually harmless, again, provided consistency.

OP, let's try to get you a recovery shell.

Start the PC, boot grub on the Tumbleweed drive. Press 'e' to edit the command line for 6.6.3. Change

Code:

root=UUID=287a2f71-d2d0-45d8-b300-82a19b18f1aa

to

Code:

root=tmpfs

This should get you a recovery shell. We won't have logs unfortunately, but we'll be able to check some stuff.

Now check /dev, look for your device. Either /dev/sdc4 or /dev/disk/by-uuid/xxxxxxxx. If the module's loading from the initrd, they'll be there.

If your device does show up, try to mount it somewhere.

Code:

mkdir /mnt
mount /dev/<whatever> /mnt

And report any error messages.

mrmazda · 12-10-2023, 12:23 AM

Quote:

Originally Posted by lattimro

wouldn't enter in a panic state if something will be wrong with initrd?

I don't think I understand this question.

Quote:

I believe at this point, for some reasons it can't find the root partition UUID=287blabla.

As I.

Quote:

Could be a fonts mistmach issue somewhere even when everything seems all right (grub.cfg)?

I can't imagine how fonts could have anything to due with this.

Quote:

As long as my outputs regarded Geckolinux posted here are from chroot environment how can one be sure what is in the native (Geckolinux) installation?

In a thread this long with so many bits supplied in separate replies, it's a tough job. It's much easier for me at least if all relevant bits are collected in one uninterrupted place, without quotes or questions, assuming that all the relevant bits have even been determined. For me in a non-booting "can't find" situation that would include at the least:

installed system's fstab
lsblk -f
fdisk -l and/or parted -l
efibootmgr -v
applicable lines from dmesg from a failing boot if possible to acquire, or a whole dmesg attached or pastebinned
anything else required to clarify exactly what OSes are installed, and where

Not being a GeckoLinux user myself, I can only wonder why at this time a 6.3.x kernel would be newly installed. Here is some selected Tumbleweed/Slowroll history:

Code:

-rw-r--r-- 1 177M Feb 16  2023 kernel-default-6.1.12-1.1.x86_64.rpm
-rw-r--r-- 1 160M Apr 22  2023 kernel-default-6.2.12-1.1.x86_64.rpm
-rw-r--r-- 1 160M Jun 24 15:35 kernel-default-6.3.9-1.1.x86_64.rpm
-rw-r--r-- 1 163M Sep 13 10:03 kernel-default-6.4.12-1.13.x86_64.rpm
-rw-r--r-- 1 162M Oct 31 19:18 kernel-default-6.5.9-1.3.x86_64.rpm
-rw-r--r-- 1 163M Nov 21 16:27 kernel-default-6.6.2-1.1.x86_64.rpm
-rw-r--r-- 1 164M Nov 30 17:06 kernel-default-6.6.3-1.1.x86_64.rpm

A 6.3.1 kernel would have been current many months ago in TW. The tools that build the kernel's initrds have gone through a generous number of iterations since then. One in my position can only guess what the relationship is, if any, between whatever makes up the current GeckoLinux toolset and Tumbleweed's. If OP is picking and choosing what to use from Gecko repos and others from TW's, one would have to expect substantial possibility of some failure. In such a situation in Debian circles, the distro as installed would be called a Frankendebian, something many would-be helpers wouldn't even try to help with.

TW users of late have been experiencing an unusually high frequency of boot failures on upgrades from 6.5.9 to 6.6.x. Even 6.5.9 had multiple iterations. If OP is using a mixture of both, I suggest it may be worth trying the latest Slowroll kernel instead of the latest TW kernel.

goumba · 12-10-2023, 06:05 AM

Quote:

Originally Posted by mrmazda

TW users of late have been experiencing an unusually high frequency of boot failures on upgrades from 6.5.9 to 6.6.x. Even 6.5.9 had multiple iterations. If OP is using a mixture of both, I suggest it may be worth trying the latest Slowroll kernel instead of the latest TW kernel.

I did not experience this, although I'm sure there are many variables involved. For one I use btrfs, OP is using ext4.

I tried looking (quickly) at the openSuSE fora (Install/Boot), but nothing from a look at post titles grabbed my attention (found one 6.6 but it was GPU related).

The only thing I wonder is why it would prevent OP from booting to the older kernel that did work. That shouldn't have been touched, the original kernel and initrd intact.

lattimro · 12-10-2023, 11:00 AM

Quote:

Originally Posted by mrmazda

I don't think I understand this question.
As I.
I can't imagine how fonts could have anything to due with this.
In a thread this long with so many bits supplied in separate replies, it's a tough job. It's much easier for me at least if all relevant bits are collected in one uninterrupted place, without quotes or questions, assuming that all the relevant bits have even been determined. For me in a non-booting "can't find" situation that would include at the least:

installed system's fstab
lsblk -f
fdisk -l and/or parted -l
efibootmgr -v
applicable lines from dmesg from a failing boot if possible to acquire, or a whole dmesg attached or pastebinned
anything else required to clarify exactly what OSes are installed, and where

Not being a GeckoLinux user myself, I can only wonder why at this time a 6.3.x kernel would be newly installed. Here is some selected Tumbleweed/Slowroll history:

Code:

-rw-r--r-- 1 177M Feb 16  2023 kernel-default-6.1.12-1.1.x86_64.rpm
-rw-r--r-- 1 160M Apr 22  2023 kernel-default-6.2.12-1.1.x86_64.rpm
-rw-r--r-- 1 160M Jun 24 15:35 kernel-default-6.3.9-1.1.x86_64.rpm
-rw-r--r-- 1 163M Sep 13 10:03 kernel-default-6.4.12-1.13.x86_64.rpm
-rw-r--r-- 1 162M Oct 31 19:18 kernel-default-6.5.9-1.3.x86_64.rpm
-rw-r--r-- 1 163M Nov 21 16:27 kernel-default-6.6.2-1.1.x86_64.rpm
-rw-r--r-- 1 164M Nov 30 17:06 kernel-default-6.6.3-1.1.x86_64.rpm

A 6.3.1 kernel would have been current many months ago in TW. The tools that build the kernel's initrds have gone through a generous number of iterations since then. One in my position can only guess what the relationship is, if any, between whatever makes up the current GeckoLinux toolset and Tumbleweed's. If OP is picking and choosing what to use from Gecko repos and others from TW's, one would have to expect substantial possibility of some failure. In such a situation in Debian circles, the distro as installed would be called a Frankendebian, something many would-be helpers wouldn't even try to help with.

TW users of late have been experiencing an unusually high frequency of boot failures on upgrades from 6.5.9 to 6.6.x. Even 6.5.9 had multiple iterations. If OP is using a mixture of both, I suggest it may be worth trying the latest Slowroll kernel instead of the latest TW kernel.

My apologize, so sorry for typo, post #34 corrected to 6.6.3-1

lattimro · 12-10-2023, 11:17 AM

Quote:

Originally Posted by goumba

Fonts have nothing to do with it. You'd just get a warning message and possibly a display that doesn't look great.

You did, but you just don't realize it

Those startup messages in your photo are the same dmesg would display.

The odd thing here is that the problem occurs across kernel versions. OP said 6.5.9 no longer works. That shouldn't have been touched.

I also revisited that bug report. The "breaking" change occurred between 6.5.3 and 6.5.4.

Also, as far as the EFI boot partition goes, while it's not recommended, as long as OP consistently mounts the EFI partition and hides (does not mount) the other, they should be alright. OP is using the EFI boot manager to choose which partition to boot from. I've installed distros on flash drives and the installer typically ignores the existing EFI System Partition, so I end up with one internal, one on the drive. Usually harmless, again, provided consistency.

OP, let's try to get you a recovery shell.

Start the PC, boot grub on the Tumbleweed drive. Press 'e' to edit the command line for 6.6.3. Change

Code:

root=UUID=287a2f71-d2d0-45d8-b300-82a19b18f1aa

to

Code:

root=tmpfs

This should get you a recovery shell. We won't have logs unfortunately, but we'll be able to check some stuff.

Now check /dev, look for your device. Either /dev/sdc4 or /dev/disk/by-uuid/xxxxxxxx. If the module's loading from the initrd, they'll be there.

If your device does show up, try to mount it somewhere.

Code:

mkdir /mnt
mount /dev/<whatever> /mnt

And report any error messages.

Thanks! I changed to tmpfs and I got a recovery shell, enter password, OK. How to check /dev?
ls /dev does not list /sdx or /disk
I will post it...

mrmazda · 12-10-2023, 11:31 AM

Quote:

Originally Posted by goumba

The only thing I wonder is why it would prevent OP from booting to the older kernel that did work. That shouldn't have been touched, the original kernel and initrd intact.

By default, openSUSE builds a new initrd for all kernels present in /usr/lib/modules/ any time packages affecting initrd construction are upgraded, and this is not configurable. I make my initrds immutable to prevent destroying known working initrds.

mrmazda · 12-10-2023, 11:36 AM

Quote:

Originally Posted by lattimro

Thanks! I changed to tmpfs and I got a recovery shell, enter password, OK. How to check /dev?
ls /dev does not list /sdx or /disk

Does /dev/disk/ exist?

It will be helpful if you can capture /run/initramfs/rdsosreport.txt, or at least look for clues what went wrong in it yourself.

lattimro · 12-10-2023, 11:46 AM

Quote:

Originally Posted by mrmazda

Does /dev/disk/ exist?

It will be helpful if you can capture /run/initramfs/rdsosreport.txt, or at least look for clues what went wrong in it yourself.

Great tip! I will post the end of the file.

As I waiting for you guys I check a few things:

- /etc/os-release seem valid to me. Also /usr/lib/os-release and the symlink /usr/lib/initrd-os are both OK.
- the cat /usr/lib/ostree/ostree-prepare-root is all gibberish
- obviously initrd-switch-root.service is failing due to os-release as you can see in the screenshot
- journalctl -b confirms the output above

colorpurple21859 · 12-10-2023, 02:42 PM

show output of blkid

goumba · 12-10-2023, 02:45 PM

Quote:

Originally Posted by mrmazda

By default, openSUSE builds a new initrd for all kernels present in /usr/lib/modules/ any time packages affecting initrd construction are upgraded, and this is not configurable. I make my initrds immutable to prevent destroying known working initrds.

Well, damn, I never noticed. I guess I just noted the current being rebuilt as last and never needed to look further. Seems commons sense would make this a good way to break a system to me (well, obviously), but I guess if there's a binary in the initrd with a security issue, you don't want that either once it's fixed.

Quote:

Originally Posted by lattimro

- obviously initrd-switch-root.service is failing due to os-release as you can see in the screenshot

I can't be certain, but that may just be a side effect of using tmpfs for root. It is likely not related to your problem.

[Edit: I just rebooted using tmpfs and I got the exact same error for initrd-switch-root. AFAIC, confirmed unrelated. Also, I deleted os-release with no ill effect. ]

Quote:

Originally Posted by lattimro

cat /usr/lib/ostree/ostree-prepare-root is all gibberish

It's a binary.

We're not concerned with /etc. What is in /dev and /dev/disk (not /disk)?

If you have a usb flash drive, mount it and copy /run/initramfs/rdsosreport.txt to it. Post the contents (if it's a lot, use something like pastebin).

lattimro · 12-10-2023, 02:56 PM

Quote:

What is in /dev and /dev/disk (not /disk)?

there is no /dev/disk, see screenshot

If you have a use flash drive, mount it and copy /run/initramfs/rdsosreport.txt to it. Post the contents (if it's a lot, use something like pastebin).[/QUOTE]