I have Salient OS systemd arch 5.16.0-arch1-1 installed on primary hdd (works fine) and in process of switching to nvme on a pciex4 addon card. PC is legacy bios. I cannot install via live cd to nvme add-on partition it fails. It does however run after retore from hdd to nvme partition by clonezilla then tunefs the uuid and fix the fstab on nvme OS. Redo grub, redo grub on HDD which boots it. After selecting the nvme drive in grub, I continue to get the "error" device not found: (insert new NVME )UUID. Then the cursor flashes about 5 seconds followed by the normal "/dev/nvme0n1p2 clean...etc, etc." and the nvme boots up no problem. But why? The fstab on nvme drive has correct UUID, the grub.cfg looks correct uuid (also tried use partuuid for root in grub.cfg-no change) not sure what I am missing here or what file or journal PID to find clues? I also rebuilt initramfs (mkintcpio -p linux on nvme) seems to make no difference. If you would like to see a file content let me know which. If it matters the boot menu selection throwing this UUID device not found error is in the second partition of mbr nvme drive and has grub itself installed. On the first partition of the nvme M.2 is slackware current with no bootloader or grub files, no errors booting that from grub. Thanks

So, generally the PCIe riser cards are NOT supported as a bootable device by devices regardless of BIOS/UEFI, therefore why grub isn't seeing it. Normally what you'd need to do to have it work flawlessly is have some supported boot medium (SATA or onboard NMVe) with the MBR installed and the /boot partition on it, then the REST of the system could be on the riser card NVMe drive. I'm actually surprised it boots at all, to be completely honest, given that it sounds like the entire system is on there, with no /boot on a still internal drive.

This smells like it could still be a UUID problem somewhere, in spite of all your reported activity. Initrds contain a UUID of the / filesystem when it was created. You may still need to regenerate the initrd and afterward grub.cfg or custom.cfg as appropriate if there was any cloning involved in preparing the NVME.

Try using root=LABEL on the kernel command line instead of root=UUID to see if it changes behavior.

Does your motherboard and chipset actually support booting from NVME on a PCIe card? What models are they?

The motherboard is a Rampage III X58 ICH10R/Marvell Sata. It's got a W3960 overclocked with lots of ram, so I am not upgrading it for this. It's fast.
Thanks for the advice. I will run a test with a throwaway drive then clone it and keep the UUID intact. I am going to dig my heels in and ride it out a while since I have a good backup and the speed is nice. I think it may not pan out and I will ultimately move the boot to a sata SSD, time will tell. I tried running a debug on a truncated grub cfg, to watch what's going on, but debug=all is too much to see anything during the menu selection, and I don't know what names to put into the debug to narrow it. Docs are thin on that info. The actual drive boot logs are clean. Something is happening in the changeover and I think you are probably right about the UUID being left behind.

Nothing anywhere near that old supports booting directly from NVME. You'll need some other storage device from which to initiate boot.

Thanks, the issue is resolved now. I removed the feature platform search hints section. Root fs is set by partuuid elsewhere. What's happening is grub is in fact booting the SSD initrd and switching to the nvme for the root fs. It was difficult to know since the nvme is a clone.

I recently encountered a similar UUID problem. My drive had free space in front of the main LVM partition and /boot partition in the middle of the drive with free space after, so I could not expand the LVM into the remaining space.

I copied everything in the /boot partition into a new partition in front of the LVM partition then in order to boot properly and not have to mess with generating a new initrd image I simply recorded the UUID info from the original /boot, wiped out that by using tune2fs and creating a random UUID for the original /boot, and then assigned the original /boot UUID to the new partition and was done. It booted properly from the proper (new) /boot partition and I was then able to finish adjusting sizes as needed.