I've recently made a LOT of progress with the board (Firefly ITX-3588J) that I mentioned over in the Embedded & SBC forum, to which I didn't get many responses to my inquiries - and so I'm not sure if this is the right forum or not for this because while it involves the board it is more specifically now related to the kernel itself. I managed to get an Ubuntu 20.04 system loaded after much trial and error because I wanted to install to an SSD hard disk, but the kernel modules don't load! In particular, running an lsmod from the command prompt gives

... and that's it. Despite that I have a /lib/modules folder populated with lots of module files and a suitable modules.dep. This means a lot of device drivers for the system are not available. And I can't bring up a graphical desktop either when it's in this state.

For what it's worth, the way I have it set up is that the kernels are on a partition on the onboard flash MMC chip, while the OS root file system is located on the hard disk. The kernel boot options are (via cat /proc/cmdline), with the root file system indicated explicitly,

The modules (in /lib/modules) are also part of the root file system, so on the hard drive, which may not be ideal - and I note that while the kernel is booting the following flashes up for a moment on the serial console output:

And note this: "Warning: Something odd, no /lib/modules." And I suspect that is what is where it is failing. What could be causing this?

FWIW, the board vendor supplies a kernel config and kernel source tree and that's what I used to compile the kernel installed on the machine. My suspicion is that it has something to do with that it needs to know at this point to look on the hard drive for the modules, yet doesn't know to do so or how to. How can you tell it?

First thing to do: Your kernel, modules,initrd, and kernel headers should all be of the same kernel version. Further, the symlinks at /lib/modules/<version>/build & /lib/modules/<version>/source should not be broken.

To quote the fictional Jean-Luc Pickard: "Make it so" . It sounds like your initrd has the wrong modules for your kernel.

All the components are the right version because I generated them all from the provided kernel code pack (5.10.66), and prepared the modules using make modules_install. Moreover, I can modprobe any module in /lib/modules/5.10.66 into the running kernel. It seems the initrd, then, must or may be the source of the problem. I'm using the "stock" initrd, i.e. the one that shipped with the same kernel code pack, so again a version incompatibility should not exist. I suppose the problem is that the stuff in the initrd doesn't know to look on a different medium (the hard drive) from the one it lives/is booted on (the eMMC) to procure the modules. How/what do I need to change in there to give it a heads up?

(FWIW it's worth noting that I actually can't use any other kernel version than this 5.10.66 right now, because the RK3588 is not a fully supported SoC in the mainline kernel code streams, particularly given how new it is. Apparently even the whole RK35xx series generally is not fully supported.)

You load a kernel & initrd at startup. They have to be the same. If you switch kernel, you hace to remake the initrd.There's usually a script called 'mkinitrd' or 'mkinitrd.sh' in the $PATH.

It might be easiest to put back the original kernel, boot on that and the mkinitrd scripts have a switch that allows you to make an initrd for a different kernel version. Try mkinitrd --help.

Is that the $PATH on the host machine (the one where I'm building the kernels) or the target one? Is it included with the kernel, or what?

ADD: I just ran it on the target machine (i.e. the Firefly/Rockchip machine) as that makes the most sense because of the architectural difference. I got the following two warnings though while it did its making:

Is this a problem?

ADD 2: I just booted it from the new initramfs. The modules still don't load - lsmod still shows only one loaded module:

Though admittedly, no user software but bash via the serial console is running at this point (X doesn't seem to be working for some reason and I have no framebuffer console either - though presumably there should be a way at least to get the former because the board's stock Ubuntu image can run X), and I've heard kernel modules are usually loaded/unloaded dynamically on a need basis. This module looks to be a wifi module. Is the system thus actually set up properly now?

FWIW now the part of the bootlog that I quoted in my initial post shows

No complaints about /lib/modules not being found or similar errors there or later. Taking it apart, it looks like actually the initramfs I was using before was the one the board maker provided which looked to be peculiar to that stock Ubuntu image and had some rather odd bits of functionality like using overlayfs - rather Android-like in that regard. Can I guess at least this issue is solved now, even though no other modules are showing as actually loaded by lsmod?

Yeah, it's slow going.

Good initrd scripts sniff around and add in the modules you want. Lesser ones either

• stick in every conceivable module or
• stick in basically no modules.

It looks like yours sticks in no modules. There's an option (often -m) to allow you to add in modules manually. I would add in every module you can think of - sound, graphics, filesystem, chipset, whatever your little thing uses. Then try it.

The errors you quoted I would ignore. Somebody compiled raid arrays into the kernel like your little sbc was a server serving the world. Raid arrays are multiple hard disks masquerading as one for various purposes (speed, redundancy, etc).

Thanks. If I open up the initrd and start poking around inside it, where is the "proper" place to put the code to add the modules? (I suppose it must be made to do modprobe with the modules.) Also, this makes sense - start with all of them, then if you find it's taking up too much RAM or CPU, start cutting back.

ADD: I just did some searching and found I can put it all in /etc/initramfs-tools/modules. Used & trimmed a find result on /lib/modules/5.10.66/kernel and put it all in there. Now it has all the modules loaded in. Not getting me graphics though, so no idea what's up there, but at least the issue in this thread is "fixed" for now, I'll say.

Thanks! I'm marking this as a solved matter now.

Run mkinitrd --help for program switches. They are usually bash scripts. Often 'mkinitrd -c' clears the tree and I'd recommend that as you've changed kernel versions. The -m option adds modules to your new initrd. All you actually need is enough to be able to mount and read the / drive.

The way this starts is that the initrd is mounted on /. That gives you the kernel modules, an 'emergency shell' which is usually busybox. When the root partition is mounted RO, it is mounted over the initrd which becomes invisible. If you ever want to use that emergency shell you need enough to make the keyboard talk to it.

Don't worry about fancy stuff or dependencies. Bells & whistles are loaded later from / and dependencies are catered for automatically.

The system does not have an "mkinitrd", but "mkinitramfs". That is what I ended up using. That does not support the same switches. Particularly, no -m switch. Also, it can mount / anyways because I compiled the kernel with the SATA drivers as integrated and not as modules for exactly that reason. The problem was that no other modules than the Broadcom module (which I believe is for the onboard wifi) were coming up. mkinitramfs does not allow dropping modules on the command line - instead, apparently you must list them in /etc/initramfs-tools/modules, and that's exactly what I did.

And I presume you mean "make the serial input talk to it", because getting a graphical console (thus keyboard) to work on this machine is a whole 'nother adventure in itself and to which I alluded on another thread on the main Software forum, as the RK3588 has a ARM Mali G610 integrated GPU, and no open source driver for that appears to exist yet, so I have to try to see what can be wrung from the closed drivers provided with the board. But I already have serial communication in all cases - though yes, over the longer term getting an independent graphical console would be a good thing and I was actually hoping that maybe some of those unloaded modules would help. Unfortunately, even with all modules loaded, however, graphics still doesn't work - not even Xorg, much less the framebuffer console for which there is no kernel config option at all for a Mali FB driver and including VGA doesn't help either (I presume ARM Mali is not VGA/PC-compatible, then, and cannot be driven in the same way a PC GPU card could be). Though for further discussion of that issue, I'd suggest looking and posting on my Graphics thread: https://www.linuxquestions.org/quest...ng-4175716561/

No I didn't mean serial port. I somehow fancied that if you dropped out of the initramfs without mounting /, you get dumped in a shell. But if you can't mount /, the only shell is in busybox. Your only comms would be the keyboard.

But you seem a bit lower down the food chain than I imagined. I think we can call this solved as if you're loading / it can get at it's own module tree.Enjoy your Graphics thread.

Yeah, thanks. Yes.