Penguin

NFSRoot is the term used to describe running an operating system with its root filesystem (/) mounted via NFS. Why would anyone do this? Thin-clients, kiosks, industrial uses, and so on. If you dont need to touch a local harddisk, you can save a lot in the way of moving parts.

oneSIS is a package that makes running NFSRoot very easy and has excellent documentation.

Other places to look for similar information include the DisklessWorkstationNotes page, and the notes on PXE and so forth.

This set up uses PXE booting network cards, via PXELinux?, to load a kernel and then mount / via NFS. Its very debian oriented, although the principles could be adapted to any situation.

Things you'll need:

  • a DHCP server, preferably ISC DHCP version 3 or later. (dhcpd3-server in woody)
  • a TFTP server that supports PXE. (tftpd-hpa in woody)
  • PXELinux? set up on your server.

This also makes use of the debian package 'diskless' to create the NFS root filesystem. If you dont run debian you'll need to find another way to make a nfsroot fs suitable for your distribution.

Procedure:

Kernel Compilation

Compile a kernel that is suitable for booting via NFS. In particular, you'll need to make sure that the kernel supports IP level autoconfiguration and mounting NFS root. Also, its probably a good idea to compile in the drivers for your network cards, as opposed to leaving them as modules.

CONFIG_NET=y CONFIG_UNIX=y CONFIG_INET=y CONFIG_IP_PNP_BOOTP=y CONFIG_IP_PNP_RARP=y CONFIG_NFS_FS=y

  1. CONFIG_NFS_V3 is not set

CONFIG_ROOT_NFS=y

  1. CONFIG_NFSD is not set

CONFIG_SUNRPC=y CONFIG_LOCKD=y

You can either compile this by hand or use make-kpkg under debian to install it, but however you do it you'll need to be able to install it into the nfs root we'll create in a second.

I also compiled my kernel with devfs support, and automount devfs on boot ticked.

Set up the root fs

Because I'm using the 'diskless' package under debian, this bit is fairly easy. diskless comes with a program called 'diskless-createbasetgz' which creates a base.tgz (potato-install style) from a basic system root. It uses debootstrap or similar to download the required packages

diskless-createbasetgz /tmp/nfsroot/ woody http://ftp.nz.debian.org/debian /tmp/base.tgz

This creates a new system under /tmp/nfsroot, from the specified mirror. It then tars this up into /tmp/base.tgz

If you are using diskless, you'll also need to download a copy of diskless-image-simple. NOTE: do NOT install this on your server. Download it manually from packages.debian.org. Copy the deb to /tmp (or to the same dir your base.tgz is in)

Now, if you run diskless-newimage it will create a new diskless group image in the directory you specify. You must run this program from a directory containing base.tgz and diskless-image-simple

  1. cd /tmp
  2. ls -l

total 56192

  • rw-r--r-- 1 root root 55278350 May 11 12:49 base.tgz
  • rw-r--r-- 1 root root 13742 May 11 12:32 diskless-image-simple_0.3.17_all.deb

drwxr-xr-x 19 root root 4096 May 11 12:48 nfsroot

Now, run diskless-newimage

  1. diskless-newimage

Follow the instructions - the defaults are probably good enough

NOTE: if you want to add things to your diskless image first, this is a good time to do it

chroot /var/lib/diskless/default/root

This puts you in a chroot jail inside the diskless image. You can use apt-get update and apt-get install to add new packages. EG, I added devfsd support here.

As this is effectivly a debian install which has only done the first reboot, you may wish to finish the install. Inside the chroot run /usr/sbin/base-config and run as usual.

Also, if you want to change the template files for /etc on your diskless machines, you can do that here. Edit /usr/lib/diskless-image/template/etc/<file> inside the chroot appropriately, before you run diskless-newhost Note: the templates reside in the above path inside the chroot, or else look in /var/lib/diskless/default/root/usr/lib/diskless-image/template/etc/<file>

You now have a basic nfsroot filesystem. Now you must create a newhost using this image (this allows you to have per-host settings for hostname etc). Use
  1. diskless-newhost /path/to/deskless/root <ip address of host>

Set up the kernel for your nfsroot

If you made a debian kernel-package, copy it to /var/lib/diskless/default/root/root (or wherever you put your default image), then chroot to /var/lib/diskless/default/root, and run dpkg -i root/kernel-image.....

You'll also need to copy the kernel image to /tftpboot/. Name it something useful. I called it 'nfsroot-kernel'

Set up TFTP

Make a directory for the tftp root dir. I usually use /tftpboot

Under debian, make sure you have the following entries in /etc/inetd.conf
tftp dgram udp wait root /usr/sbin/in.tftpd -p -s /tftpboot

This tells tftp to use /tftpboot as its root - all paths are now relative to this.

Set up DHCP

Make sure your dhcpd config file contains something like (for dhcp 2.x, default in Debian Woody)

allow bootp; allow booting;

authoritative; ... host nfsroot {

hardware ethernet 00:01:02:03:04:05; fixed-address nfsroot.mydomain.tla; filename "pxelinux.0";

}

This tells the client with the mac address 00:01:02:03:04:05 to use the IP address associated with the name nfsroot.mydomain.tla (this only works if you have your DNS set up correctly. You can specify an IP address here instead of a name if you dont have DNS working). It then loads the file pxelinux.0, which is its bootloader.

Set up PXE

Copy pxelinux.0 from the pxelinux distribution under /tftpboot

Make the directory /tftpboot/pxelinux.cfg - this is where you'll store all your configurations.

In /tftpboot/pxelinux.cfg create a file called 'default', and add something like the following to it:

/tftpboot/pxelinux.cfg/default

default nfs

label nfs

kernel netboot-kernel append root=/dev/nfs \

nfsroot=10.66.1.1:/var/lib/diskless/default/root/ \ ip=dhcp

** IMPORTANT: concatenate the 'append' line, and remove the \ characters

Note, that the default file is the the default configuration file for the client. PXE actually searches first of all for its IP address printed in hexadecimal, then drops a byte off, and continues looking for this until it has to give up and use the default file. This lets you add a config file for each host you boot via PXE.

This pxe config file sets the default boot option to nfs. It tells it to boot the kernel 'netboot-kernel' which it gets via tftp. It appends the specified kernel options to the kernel. These options are all, as far as I can tell, ABSOLUTELY required.

Note: you can specify NFS mount options via the nfsroot kernel parameter. Simply append them after the root dir. The syntax is nfsroot=?<root-dir>[,<nfs-options>?. You will need this if you want to mount the filesystem using NFS v3 which is needed if you want to work with large files. (In NFSv2, the default for Linux clients, there is a 2GB filesize limit.). To enable NFSv3 use the "v3" option.

Set up NFS

You'll need to add something like the following lines to /etc/exports

  1. per-host diskless exports

/var/lib/diskless/default/10.66.1.104 10.66.1.104(rw,no_root_squash)

  1. global diskless exports

/home 10.66.1.0/255.255.255.0(rw,no_root_squash) /var/lib/diskless/default/root 10.66.1.0/255.255.255.0(rw,no_root_squash)

Note that these are different from the suggested options in the diskless package. I found mounting NFS root ro caused a lot problems.

Restart your nfs server (/etc/init.d/nfs-kernel-server restart in debian) and you are away

Booting the client

Set up your client to boot via PXE, either by using a PXE enabled boot card or using the pxe-on-disk image you can find under PXE. This only works with certain cards tho - intel eepro, 3com, realtek 8139, netgear (natsemi chipset), and some others.

This should all go!

Cavaets

  • I found that createbasetgz didn't do a very good job of it. I had to chroot to /var/lib/diskless/default/root and run apt-get -f install to tidy it up.
  • If you still have problems with logging in, and you see init complain about a read-only root fs, you might need to edit the fstab file of each client. Look in /var/lib/diskless/default/<client-ip>/etc and change the 'ro' options in /etc/fstab to 'rw'. You can set this for all new clients by editing /var/lib/diskless/default/root/usr/lib/diskless-image/template/etc/fstab and changing it there.
  • Re-running diskless-newhost on an IP that you've already used will update its /etc configuration.