Diskless RHEL3

Lexical Unit · 10-09-2006, 01:24 PM

Hello netbooting gurus,

Thank you in advance for any help, I really need it.

Short version:
mount: can't find / in /etc/fstab or /etc/mtab errors in initrd's linuxrc script on a diskless client.

Details:
I'm in the process of getting a bunch of RHEL3 systems to netboot off one central NFS/DHCP/TFTP server.

The bulk of my information on how to do this has come from www_intra2net_com/de/produkte/opensource/diskless-howto/howto.html
and www_redhat_com/mirrors/LDP/HOWTO/Diskless-root-NFS-other-HOWTO.html#toc4 as well as other various sites found on google. But mostly the first source. (Um, I'm sorry if I'm breaking a rule by posting a URL... but not listing my sources would be grossly detrimental to the coherency of this post, and -- I feel -- needlessly so. Replace _ with . of course)

I've got everything working right up to the point where the initrd image is loaded and the linuxrc script begins to execute.

The script has a problem trying to remount / as read-write. Here is all the relevant information I can think of:

NFS exports:

Code:

/diskless/root/                  *(ro,async,no_root_squash)

dhcpd.conf:

Code:

... the usual ...
next-server <ip of NFS server>;
option root-path "<ip of NFS server>:/diskless/root/";

/tftpboot/pxelinux.cfg/default:

Code:

prompt 1
default test
timeout 100

label test
kernel vmlinuz-2.4.21-27.ELsmp
append init=/linuxrc root=/dev/ram0 initrd=initrd.img.gz ramdisk_size=204080 noapic acpi=off

Now a quick run though of what's inside initrd.img.gz.
/bin is setup with Busybox v1.00-pre4 for most utils except the following:
* find (b/c busybox does't support -exec for find)
* mount (because I want a more verbose mount)
* pivot_root (because I don't trust busybox to do it right)

Now let's have a look at the linuxrc file:

Code:

#!/bin/sh
echo "Remount / read-write"
mount -v -n -o remount,rw / && echo "Worked!"
... other stuff, the line above fails so I'm leaving it off till that problem is solved ...

Ok, so. Here's what happens when I boot my diskless client.

PXE gets IP from DHCP server.
TFTP sends over vmlinuz and initrd
The kernel uncompresses and does it's typical stuff.
We start to execute the linuxrc script.
And we get:

Code:

warning: can't open /etc/fstab: no such file or directory
warning: can't open /etc/mtab: no such file or directory
mount: can't find / in /etc/fstab or /etc/mtab
...
Eventually kernel panics b/c it can't write to a file in /etc

During bootup "/" is a tmpfs based filesystem until the boot script switches to the real root. There's no reason it shouldn't be read-write and remounting it as read-write shouldn't be a problem.

Anyone have any ideas here?

Lexical Unit · 10-10-2006, 09:52 AM

I had an idea to get some better informaiton about what was going on, maybe this will help figure things out.

1. We know everything up to the init script works perfectly.
2. The first thing the init script needs to do is remount root as rw so that it can do it's thing.

Here's the linuxrc script for reference:

Code:

echo "Remount / read-write"
mount -n -o remount,rw / && echo "Worked!"
sleep 10

echo "Mounting /proc filesystem"
mount -n -t proc /proc /proc
cat /proc/mounts
sleep 10

echo "Load NIC module"
insmod tg3

echo "Loading nfs modules"
modprobe nfs

echo "Initializing network loopback device"
ifconfig lo 127.0.0.1 up
    
echo "Configuring eth0"
udhcpc --now --quit --interface=eth0 --script=/bin/udhcpc.script
if [ $? -ne 0 ]; then
    echo "ERROR: couldn't get DHCP lease, trying again"
    udhcpc --now --quit --interface=eth0 --script=/bin/udhcpc.script
    if [ $? -ne 0 ]; then
        echo "ERROR: couldn't get DHCP lease, trying again"
        udhcpc --now --quit --interface=eth0 --script=/bin/udhcpc.script
        if [ $? -ne 0 ]; then
            echo "ERROR: can't get DHCP lease"
            exit 0
        fi
    fi
fi

# load DHCP parameter
. /etc/udhcpc-eth0.info

echo "Mounting nfs root filesystem"
if ! echo $ROOTPATH | grep -q ":/" ; then
    # we haven't got a full path, use next-server
    ROOTPATH="${NEXTSERVER}:${ROOTPATH}"
fi

if echo $ROOTPATH | grep -q "," ; then
    # we have options
    NFSOPTIONS=`echo $ROOTPATH | sed -e "s/\(.*\)\(,.*\)/\2/"`
    ROOTPATH=`echo $ROOTPATH | sed -e "s/\(.*\)\(,.*\)/\1/"`
fi

echo "Root-path = $ROOTPATH"
sleep 10

echo "Mounting root filesystem"
mount -rw -t tmpfs none /sysroot/

echo "Mounting NFS root-base"
mkdir /sysroot/nfsroot
mount -n -o "ro,nolock${NFSOPTIONS}" -t nfs "$ROOTPATH" /sysroot/nfsroot
if [ $? -ne 0 ]; then
    echo "ERROR: can't mount root filesystem via NFS"
    exit 0
fi

ls /sysroot/nfsroot
sleep 10

echo "Setting root symlinks"
cd /sysroot
find ./nfsroot -maxdepth 1 -mindepth 1 -exec ln -s \{\} \;
cd /

echo "Handling special directories"
rm -f /sysroot/initrd
mkdir /sysroot/initrd
rm -f /sysroot/tmp
mkdir /sysroot/tmp
rm -f /sysroot/proc
mkdir /sysroot/proc

echo "Copying /var"
rm -f /sysroot/var
mkdir /sysroot/var
cp -a /sysroot/nfsroot/var /sysroot
cp /etc/resolv.conf /sysroot/var/resolv.conf

# /dev handling
rm -f /sysroot/dev
mkdir /sysroot/dev
if [ -f /sysroot/nfsroot/dev.tar.gz ]; then
    echo "Unpacking /dev"
    tar xzf /sysroot/nfsroot/dev.tar.gz -C /sysroot
else
    echo "Copying /dev"
    cp -a /sysroot/nfsroot/dev /sysroot
fi

echo 0x0100 > /proc/sys/kernel/real-root-dev

echo "Unmounting temporary mounts"
umount /proc

echo "Changing to new NFS root"
pivot_root /sysroot /sysroot/initrd && echo "Worked!"

Please just ignore the longness of it, the really important parts are bolded.

Now, since mount was complaining about not finding fstab, I put a fstab into the ramdisk at /etc/fstab with the following contents:
none / tmpfs defaults 0 0
I HAVE NO IDEA IF THIS IS CORRECT

Now let's start the diskless client and see what we get:
1. The remount of / as read-write line prints "Worked!"
2. The cat of /proc/mounts works and gives:
rootfs / rootfs rw 0 0
/dev/root / ext2 rw 0 0
/proc /proc proc rw 0 0
3. The echo of root-path gives me exactly what I want, the ip of the NFS server followed by :/path/to/nfs/export/root/

4. After I mount the nfs root, ls'ing it gives exactly what is on the NFS server. So yay, the nfs mount works perfectly.

5. The pivot_root line prints "Worked!"

And then, directly after the "Worked!" from the pivot_root line, I get:
Kernel panic: Attempted to kill init!

Going back, I delete the /etc/fstab from the initrd and try again:
1. The remount line does not echo "Worked!" (errors given related to not finding fstab or mtab)
2. rootfs / rootfs rw 0 0
/dev/root / ext2 ro 0 0

/proc /proc proc rw 0 0
3. kernel panic before I get here because udhcpc can not create a file on a read only file system

So, uh, does any of that give anyone some ideas?

Lexical Unit · 10-13-2006, 10:29 AM

I got the minimal install to work by using the hand-crafted fstab in the initrd and then adding the line

exec /sbin/init

to the end of the linuxrc script so that the initrd init can hand control over to the real init. Works like a charm.

Now I'm up against a rather mysterious problem.

Since I have things working with a minimal install of just a handful of rpms. I wanted to then move up a step and netboot a fully installed RHEL3 system.

I have a fully installed RHEL3 system, and I just rsync'ed it (excluding /proc, /dev, and /diskless of course) to /diskless/me/root.

Then had it export /diskless/me/root, changed DHCP to look for that rootpath, fixed up the /diskless/me/root/etc/fstab file to be diskless-friendly, and attempted to netboot the diskless client.

Everything worked just fine. My linuxrc init script executed just like before, with no problems at all. Modules load, filesystems mount, nfs mounted, pivot_root works.

Then it gets to the exec /sbin/init line and that seems to work, but then it hangs. No output, no echoing of keys, no nothing. It just hangs forever. I can replace the exec init line with /bin/bash and that works just fine. But calling exec /sbin/init from bash causes the same infinite hang.

Any ideas?