The delays have ranged from as little as ~90 seconds, but are more typically in the 3-4 minute range, and occasionally take more than 13 minutes. This started happening more than two years ago, around the time dracut took hold replacing mkinitrd. It was only about that time that I first used EXT4 instead of EXT3. Older installations using mkinitrd have never produced these delays. Also, these delays only occur with 64 bit kernels. My first attempt to get help was probably http://www.spinics.net/lists/linux-i.../msg04173.html which produced no fruit. A follow-up to https://lists.opensuse.org/opensuse-.../msg00028.html did little better.

I've been multibooting multiple PCs since last century, so I'm quite familiar with Grub setup and configuration, on most PCs booting most of the time from openSUSE's Grub, which I install manually on an EXT2 primary partition on virtually every boot device. A few are not on primaries, with the primary bootloader on those being IBM Boot Manager from OS/2. Most of these primary Grubs are using 13.1's version 0.97-194.1.2, and most setups were initialized running the 32 bit version. All HDs have generic code in their MBRs. I don't allow Grub2 to be installed anywhere except on Kubuntu, which here sees little uptime at all, and almost never gets booted by its own bootloader. Distros other than Kubuntu that do not offer Grub get installed sans native bootloader, after which I manually setup openSUSE's Grub on them. This problem applies to multiple distros, including Debian, Fedora, Mageia and openSUSE, including testing versions of each except Mageia. openSUSE is on my 24/7 Linux PC, so it sees more uptime than all other distros combined, yet has never exhibited any boot delay.

Because I'm multibooting, I try best I can to main as much compat as possible as kernels, filesystems and drivers evolve. To that end, -I128 is always used and -b1024 on all disks lacking k4 sectors internally when formatting both EXT3 and EXT4. I partition and format in advance of each and every OS installation. The only common exception to advance formatting is that the Debian installer always insists on formatting swap, which is very inconvenient in multiboot, but I work around that by not enabling swap in Debian installations until after initial boot.

These delays are always initrd specific. IOW, a given initrd will be delayed either every time, or never. The amount of delay is also specific to any given initrd. The delays don't depend on use of my primary bootloader, as I can chainload to the bootloader on the target and the delay will still occur with a given initrd. Until any given initrd is created, there seems to be little reliable expectation whether it will be a failure or normal, though typically once any one occurs, the next created for the same or next kernels on that installation is more likely to exhibit the delay than not.

IIRC, these delays have been limited to three PCs, one HD installed to each of hosts big31 and big41, two to host mcp61. It may also have occurred on a fourth and possibly a fifth, neither of which I can recall. Checking with smartctl for possible HD problems produced no such indications.

I manage Grub well enough that I can't remember the last situation asking for help with it, and have no idea how to troubleshoot what's going on between boot stanza selection and kernel taking over. I can avoid the delay with certainty in one of only two ways so far discovered:

1-load the initrd from an alternative EXT3 filesystem, or

2-backup the entire / filesystem, format / EXT3, restore

Both work equally effectively, but are similarly cumbersome to create or maintain.

I suspect the problem is something in EXT4 evolution that never got any dev attention in conjuction with Grub Legacy as bootloader, but cannot rule out Dracut as cause.

http://fm.no-ip.com/Share/Linux/ has 5 relatively recently created initrds that produce delays, each from a different installation.

Can anyone identify a way to find the root cause of and eliminate these delays?

I would be inclined to agree - and I suspect the ext4 devs neither know about it nor care. Legitimately IMHO.
Grub legacy is just that - unmaintained and unloved. If it breaks, you (that's a royal plural, not you specifically BTW) get to keep all the pieces.
I don't expect you'll get a lot of sympathy.If it hurts, stop banging your head on that brick wall. Either don't use ext4, or (better) move on and use something current as your loader. Grub, both legacy and grub2, have been able to boot logical partitions for many years - grub2 has many other benefits despite its learning curve.

Clearly. Nevertheless, I did see something last year or before that I haven't been able to remember details about or find again about one new or changed optional EXT4 feature that might be the root problem, and wish I could identify what that might be.
I agree that officially this is true, but in openSUSE it has 8 changelog entries from 2013 and two each from 2014 and 2015, so unofficially it looks at least quasi-maintained.
I've been converting back to EXT3 on installations where the delay has been encountered, but it's very time consuming. If I convert all, then I have no practical way to follow-up with any bug report or help request's offered solution.
I fail to find any point in this comment. My average installation count per HD is well in excess of 10. With very few exceptions, all have bootloaders installed, mostly Grub, but they are used almost exclusively for those few times I wish to boot a kernel older than the current version. I've had logicals booting from their own Grubs for well over a decade, and from IBM BM more than two decades.
Grub2, besides bloat and learning curve, comes with other disadvantages:
1-functional dependence on scripts
2-functional dependence on mounting of setup targets
3-repeated complaining about supposed unreliability installed using blocklists
4-repeated complaining about multiple numberless initrd and vmlinuz symlinks that facilitate multiboot using a master bootloader
5-components scattered among several locations
6-incompatibility with 0.9x, with two different device referencing specifications dependent on context
7-extremely difficult to manage boot menu configuration manually
8-functionally incompatible with use on a partition not mounted to /boot (master bootloader on primary not managed by an OS)
9-awkward to have it depend on humanly comprehensible labels instead of robots and eidetics only UUIDs.


It's hard to remember all the drawbacks, while using Grub is not only vastly simpler (e.g. setup can be run from its shell against any partition regardless of what is or is not mounted, a major convenience in a a multiboot environment featuring frequent cloning), in its openSUSE gfxboot incarnation it is both beautiful and exceptionally convenient for making cmdline alterations on the fly.

Thank you for replying syg00.

I read this as meaning you thought grub (legacy) lacked that facility.

As for grub, gpt was the last straw for me - I've never liked the verbosity of grub2 (comes from it's mainly Ubuntu/Debian based devs), but it solves all the problems I need to confront. The complexity (at least some of it) becomes a necessary evil once that design is accepted.

I've only ever experienced this delay on one machine, only when booting from USB. I don't I've tried it with any other filesystem besides ext4. I no longer have the machine for testing though.

I've had a similar issue but it wasn't Grub legacy related (although that is what I also use).
After altering partitions either to change the layout or often simply by formatting them for a new distro (I multiboot), I sometimes get up to a 2 min wait for a distro to load. The issue was with the uuids being changed and the initrd not being able to find the system so the wait was for the system to time out and revert back to the old /dev/sdXX system. I always fixed it by creating a new initrd.
This doesn't sound like the issue you are having but it couldn't hurt to make a new initrd and see if it helps.

The limitation to few machines, 64-bits only, dracut built initrds, EXT4 / filesystem, and floppy-equipped, seems to have been BIOS-related. BIOS menus aren't necessarily clear which option(s) enable AHCI or not. Making BIOS changes to ATA got rid of the worst delays on the two machines producing long delays routinely. Now instead of delays measured in minutes, the worst are no more than 10-12 seconds.

It turns out this hadn't left every machine after all. On host big31 it returned with random kernels. With the last newly built initrd, for kernel 4.4.126 on openSUSE 42.3, the whole time to boot from Grub selection into shell prompt appearance repeatedly came to about 145 seconds. I made a copy of its initrd, then made a copy of the copy, then replaced the original with the second copy. That process reduced total boot time to roughly 30 seconds.

Is it possible for you to test this on partition which starts @64 and partition which starts @2048 to measure any difference with the delays?
I ask because I had something similar on BIOS machines with vFAT format USB drives with syslinux, there were long delays if bootable partition started @2048.

All mine are multiboot PCs. No / partition is on a primary. Big31's first partition is @64, a 400MB EXT2 used for booting but never mounted to any /boot directory. Grub setup on it likely was done booted to Knoppix. Since Debian's Grub 0.97 is known to have trouble with EXT4, I've just rerun setup on it using openSUSE 13.1's Grub 0.97-194, then booted from it using a configfile stanza an openSUSE 4.4.104 kernel/initrd that I had recorded as a delayed booter, with no apparent delay. Time will tell if other delayed boots recur.

I still haven't identified the problem, but I do seem to have found a key puzzle piece that can be used to avoid the problem. Thinking that there must be some EXT4 filesystem feature default that must have been changed around 3 years ago, first I cloned the host big31 HD to another, formatted the 8 problem partitions with mkfs.ext4 1.41.14 booted to openSUSE 11.4, then rsync'd from the cloned partitions back to the freshly formatted partitions. After so doing, all boot delays disappeared. I was elated.

Then I did updates that included new kernels on three installations. The delays returned when booting the new kernels, stayed gone booting prior kernels. I was very puzzled.

The two big producers of the problem have virtually identical AMI BIOS from late 2009/early 2010 on motherboards from Biostar. It turns out that 3 levels deep in SATA device configuration are defaults to 32Bit Data Transfer disabled and DMA Mode auto, settings I can't recall seeing in any other PC BIOS. I changed DMA to UDMA6 and enabled 32bit DT.

First boot after the BIOS changes using one of the new kernels, 4.18.5 on Fedora 29, retained some delay. Then I regenerated the 4.18.5 initrd and rebooted to find delay virtually gone. On rebooting Buster and one other (I forgot which already) to their fresh kernels, delay was already entirely gone without regenerating new initrds for their new kernels.

So, at this point it appears there is some inexplicable interplay between these particular BIOS settings and whatever it is that Dracut does to build initrds. I doubt I'll be doing any more experiments now that I know the delays are easily avoided via those BIOS settings.

I'm not sure about other distros but opensuse, in grub2, has a grub.cfg section 90_persistent. You can create a 'menu.lst' type entry here and it is copied as-is for each update.
If you remove the executable from all the /default/grub entries ex 00_header and 90_persistent you're left with a 'menu.lst'.
You do have to amend it each time there's a grub, X, or kernel update but otherwise it's like using grub legacy with the benefits of grub2.

edit: ...and I believe grub legacy has to be on an ext3 fs where grub2 works on ext4.

I find there to be no benefits outweighing the bloat of grub2, so only use only Gfxboot Grub Legacy on MBR. On GPT/UEFI I use grubx64.efi and 07_custom with a UUID-free custom.cfg that's a small fraction of the size of generated grub.cfgs and only needs updating when adding or removing an installed OS. e.g.The EXT3 limitation doesn't apply to openSUSE's Grub Legacy when EXT4 formatting includes -O '^64bit'. I find 64bit pointless on smallish partitions used for /boot and / on MBR disks, and contra-indicated on the disks that retain older installations such as 11.4 and back as far as 10.2.

Both grub and grubefi stanzas I actually boot from use only kernel and initrd symlinks, so are never directly affected by version number changes.

Are those affected files heavily fragmented? You can use debugfs or "hdparm --fibmap" to find out. Curing that fixed the one case I encountered of slow loading, though perhaps that was a case of, "Post hoc ergo propter hoc". GRUB relies on BIOS calls for physical I/O, and loading a file in tiny pieces can be slow.

1 members found this post helpful.

I hadn't realised until I read your post #13 that we've had a similar "conversation" in a previous post. I apologise for again putting forward my simplistic suggestion.