--- /dev/null
+[[!meta title="Diskless Debian Etch"]]
+
+Caveat: I have much more Linux experience than I had when I wrote this
+HOWTO. If I had to do it again, I'd probably use [[Gentoo]] instead
+of [[Debian]], because I find it easier to write custom packages for
+Gentoo. Anyhow, here's my original HOWTO, preserved for posterity.
+
+
+Overview
+========
+
+This HOWTO details the procedure I used to set up the abax cluster for
+NFS-rooted network booting. The system is useful in this case because
+it centralizes the installation in the head node (server), which makes
+maintaining, upgrading, or altering the computational nodes (clients)
+easier.
+
+This procedure follows mainly Tim Brom's [Microwulf configuration
+notes][microwulf] with two major differences.
+
+* Microwulf uses Ubuntu (gutsy?), and I'm using Debian etch.
+* Microwulf has a seperate partition for each client's root, populated
+ with an independent installation from CD. I'm using a single
+ partition for all of my clients, with the base system created using
+ debootstrap (no CD).
+
+For guidance in my deviations, I'm indebted to Bart Trojanowski's
+[pxeboot and nfsroot notes][debian-nfsroot] and Falko Timme's notes on
+[kernel compilation in Debian Etch][kernel].
+
+
+Physical setup
+==============
+
+Our cluster has one server with eight clients. The server has two
+network cards, `eth0` and `eth1`. `eth1` is connected to the outsize
+world (WAN). All of the clients have one network card, `eth0`. All
+of the `eth0`s are connected together through a gigabit switch (LAN).
+
+
+Notation
+========
+
+Throughout this HOWTO, I will use `#` as the prompt for root, `$` as
+the prompt for an unpriveledged user, and `chroot#` as the prompt for
+a root in a `chroot`ed environment. File contents will be listed with
+the full path in the text introducing the listing. For example,
+`path/to/file`:
+
+ Contents of file
+
+All files are complete with the exception of lines containing `…`, in
+which case the meaning of the example should be clear from the
+context.
+
+
+Basic server setup
+==================
+
+Installing the OS
+-----------------
+
+Boot the server with the Debian installation kernel following one of
+the options in the [Debian installation guide][install]. I netbooted
+my server from one of the client nodes following this procedure to set
+up the DHCP and TFTP servers on the client and untarring
+[netboot.tar.gz][] in my `tftpboot` directory. After netbooting from
+a client, don't forget to take that client down so you won't have
+DHCP conflicts once you set up a DHCP server on your server.
+
+Install Debian in whatever manner seems most appropriate to you. I
+partitioned my 160 GB drive manually according to
+
+<table style="margin-left: auto; margin-right: auto">
+ <tr><th>Mount point</th><th>Type</th><th>Size</th></tr>
+ <tr><td>`/` </td><td>ext3</td><td>280 MB</td></tr>
+ <tr><td>`/usr` </td><td>ext3</td><td>20 GB</td></tr>
+ <tr><td>`/var` </td><td>ext3</td><td>20 GB</td></tr>
+ <tr><td>`/swap` </td><td>swap</td><td>1 GB</td></tr>
+ <tr><td>`/tmp` </td><td>ext3</td><td>5 GB</td></tr>
+ <tr><td>`/diskless`</td><td>ext3</td><td>20 GB</td></tr>
+ <tr><td>`/home` </td><td>ext3</td><td>93.7 GB</td></tr>
+</table>
+
+I went with a highly partitioned drive to ease mounting, since I will
+be sharing some partitions with my clients. To understand why
+partitioning is useful, see the [Partition HOWTO][partition].
+
+You can install whichever packages you like, but I went with just the
+standard set (no Desktop, Server, etc.). You can adjust your
+installation later with any of (not an exhaustive list)
+
+* [`tasksel`][tasksel], command line, coarse-grained package control.
+* `apt-get`, command line, fine-grained package control.
+* `aptitude`, curses frontend for `apt-get`.
+* `synaptic`, gtk+ frontend for `apt-get`.
+* `dpkg`, command line, package-management without dependency checking.
+
+The base install is pretty bare, but I don't need a full blown
+desktop, so I flesh out my system with:
+
+ # apt-get install xserver-xorg fluxbox fluxconf iceweasel xterm xpdf
+ # apt-get install build-essentials emacs21-nox
+
+which gives me a bare-bones graphical system (fire it up with
+`startx`) and a bunch of critical build tools (`make`, `gcc`, etc.).
+
+
+Configuring networking
+----------------------
+
+We need to set up our server so that `eth1` assumes it's appropriate static IP on the WAN, and `eth0` assumes it's appropriate static IP on the LAN.
+We achieve this by changing the default `/etc/network/interfaces` to
+
+ # This file describes the network interfaces available on your system
+ # and how to activate them. For more information, see interfaces(5).
+
+ # The loopback network interface
+ auto lo
+ iface lo inet loopback
+
+ allow-hotplug eth0 # start on boot & when plugged in
+ iface eth0 inet static # static LAN interface
+ address 192.168.2.100
+ netmask 255.255.255.0
+ broadcast 192.168.2.255
+
+ allow-hotplug eth1 # start on boot & when plugged in
+ #iface eth1 inet dhcp # WAN DHCP interface (not used)
+ iface eth1 inet static # WAN static interface
+ address XXX.XXX.YYY.YYY
+ netmask 255.255.128.0
+ broadcast XXX.XXX.127.255
+ gateway XXX.XXX.ZZZ.ZZZ
+
+where I've censored our external IPs for privacy. The netmask selects
+which addresses belong to which networks. The way we've set it up,
+all 192.168.2.xxx messages will be routed out `eth0`, and everything
+else will go through `eth1` to it's gateway. See the [Net-HOWTO][]
+for more details.
+
+
+Remote booting
+==============
+
+Server services
+---------------
+
+The clients will boot remotely using the Pre eXecution Environment
+(PXE). The boot procedure is
+
+1. Client powers on.
+2. Client BIOS comes up, detects attached devices, and looks for a
+ DHCP server for advice on network booting.
+3. DHCP server gives client an IP address, domain name, host name, the
+ IP address of the TFTP server, and the location of the bootloader
+ on the TFTP server.
+4. Client gets bootloader from TFTP server.
+5. BIOS hands over control to bootloader.
+6. Bootloader gets kernel and initial ramdisk from TFTP server.
+7. Bootloader hands over control to kernel
+8. Kernel starts up the system, mounting root via NFS.
+9. … after this point, it's just like a normal boot process.
+
+We can see that we need to set up DHCP, TFTP, and NFS servers (not
+necessarily on the same server, but they are in our case).
+
+### pxelinux
+
+The [pxe][] bootloader can be obtained with
+
+ # apt-get install syslinux
+
+which installs it to `/usr/lib/syslinux/pxelinux.0` along with a
+manual and some other `syslinux` tools.
+
+### DHCP
+
+Install a server with
+
+ # apt-get install dhcp
+
+Configure the server with `/etc/dhcpd.conf`
+
+ allow bootp; # maybe?
+ allow booting;# maybe?
+ option domain-name "your.domain.com";
+ option domain-name-servers XXX.XXX.XXX.XXX,YYY.YYY.YYY.YYY;
+
+ subnet 192.168.2.0 netmask 255.255.255.0 {
+ range 192.168.2.150 192.168.2.200; # non-static IP range
+ option broadcast-address 192.168.2.255;
+ option routers 192.168.2.100; # Gateway server
+ next-server 192.168.2.100; # TFTP server
+ filename "pxelinux.0"; # bootloader
+
+ host n1 {
+ hardware ethernet ZZ:ZZ:ZZ:ZZ:ZZ:ZZ;
+ fixed-address 192.168.2.101;
+ option root-path "192.168.2.100:/diskless/n1";
+ option host-name "n1";
+ }
+ … more hosts for other client nodes …
+ }
+
+This assigns the client a static hostname, domain name, and IP address
+according to it's ethernet address (aka MAC address). It also tells
+all the clients to ask the TFTP server on 192.168.2.100 for the
+bootloader `pxelinux.0`. For extra fun, it tells the clients to send
+packets to the router at 192.168.2.100 if they can't figure out where
+they should go, and to use particular DNS servers to resolve domain
+names to IP addresses. This gives them access to the outside WAN. I
+don't know yet if the booting options are necessary, since I don't
+know what they do.
+
+We also need to ensure that the DHCP server only binds to `eth0`, since starting a DHCP server on your WAN will make you unpopular with your ISP. You should have the following `/etc/default/dhcp`:
+
+ INTERFACES="eth0"
+
+Once the DHCP server is configured, you can start it with
+
+ # /etc/init.d/dhcp restart
+
+Check that the server is actually up with
+
+ # ps -e | grep dhcp
+
+and if it is not, look for error messages in
+
+ # grep -i dhcp /var/log/syslog
+
+
+### TFTP
+
+There are several TFTP server packages. We use `atftpd` here, but
+`tftp-hpa` is also popular. Install `atftpd` with
+
+ # apt-get install atftpd xinetd
+
+where `xinetd` is a super-server (replacing `inetd`, see `man xinetd`
+for details). Configure `atftpd` with `/etc/xinetd.d/atftpd`
+
+ service tftp
+ {
+ disable = no
+ socket_type = dgram
+ protocol = udp
+ wait = yes
+ user = nobody
+ server = /usr/sbin/in.tftpd
+ server_args = --tftpd-timeout 300 --retry-timeout 5 --bind-address 192.168.2.100 --mcast-port 1758 --mcast-addr 239.239.239.0-255 --mcast-ttl 1 --maxthread 100 --logfile /var/log/atftpd.log --verbose=10 /diskless/tftpboot
+ }
+
+Note that the `server_args` should all be on a single, long line,
+since I haven't been able to discover if `xinetd` recognizes escaped
+endlines yet. This configuration tells `xinetd` to provide TFTP
+services by running `in.tftpd` (the daemon form of `atftpd`) as user
+`nobody`. Most of the options we pass to `in.tftpd` involve
+multicasting, which I believe is only used for MTFTP (which
+`pxelinux.0` doesn't use). `--logfile /var/log/atftpd.log
+--verbose=10` logs lots of detail to `/var/log/atftpd.log` if it
+exists. You can create it with
+
+ # touch /var/log/atftpd.log
+ # chown nobody.nogroup /var/log/atftpd.log
+
+The most important argument is `/diskless/tftpboot`, which specifies
+the root of the TFTP-served filesystem (feel free to pick another
+location if you would like). This is where we'll put all the files
+that the TFTP will be serving. It needs to be read/writable by
+`nobody`, so create it with
+
+ # mkdir \tftpboot
+ # chmod 777 tftpboot
+
+(TODO: possibly set the sticky bit, remove writable?)
+
+Finally, we need to restart the `xinetd` server so it notices the new
+`atftpd` server.
+
+ # /etc/init.d/xinetd restart
+
+Check that the `xinetd` server is up with
+
+ # ps -e | grep xinetd
+
+and look for error messages with
+
+ # grep -i dhcp /var/log/syslog
+
+Just having `xinetd` up cleanly doesn't prove that `atftpd` is working
+though, it just shows that the `atftpd` configuration file wasn't too
+bungled. To actually test `atftpd` we need to wait until the
+[Synthesis Section][sec.synthesis] when we actually have files to
+test-transfer.
+
+### NFS
+
+Install the NFS utilities on the server with
+
+ # apt-get install nfs-common nfs-kernel-server
+
+We go with the kernel server because we want fast NFS, since we'll be
+doing a lot of it. Set the NFS server up to export the root file
+systems and the user's home directories with `/etc/exports`:
+
+ /diskless/n1 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
+ … other node root exports …
+ /diskless 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check) # unnecessary
+ /home 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
+ /usr 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
+
+Then let the NFS server know we've changed the `exports` file with
+
+ # exportfs -av # TODO: -r?
+
+Test that the NFS server is working properly by `ssh`ing onto one of
+the clients and running
+
+ client# mkdir /mnt/n1
+ client# mount 192.168.2.100:/diskless/n1 /mnt/n1
+ client# ls /mnt/n1
+ … some resonable contents …
+ client# umount /mnt/n1
+ client# rmdir /mnt/n1
+
+
+Client setup
+------------
+
+The only client setup that actually happens on the client is changing
+the BIOS boot order to preferentially boot from the network. Consult
+your motherboard manual for how to accomplish this. It should be
+simple once you get into the BIOS menu, which you generally do by
+pressing `del`, `F2`, `F12`, or some such early in your boot process.
+*Everything else happens on the server*.
+
+### Root file system
+
+We want to install a basic Debian setup on our clients. Since each
+client doesn't have it's own, private partition, we need to install
+Debian using `debootstrap`.
+
+ # apt-get install debootstrap
+ # mkdir /diskless/n1
+ # debootstrap --verbose --resolve-deps etch /diskless/n1
+ # chroot /diskless/n1
+ chroot# tasksel install standard
+ chroot# dpkg-reconfigure locales
+ chroot# apt-get install kernel-image-2.6-686 openssh-server nfs-client
+
+TODO: what get's installed with standard?
+See `/usr/share/tasksel/debian-tasks.desc` for a list of possible
+tasks and [the debian docs][internals] for details on how a full
+installation from CD or netboot.
+
+We can also add a few utilities so we can work in our `chroot`ed environment
+
+ chroot# apt-get install emacs21-nox
+
+### Configuring `/etc`
+
+The client will be getting its hostnames from the DHCP server, so
+remove the default
+
+ # rm /diskless/n1/etc/hostname
+
+We also need to setup the `fstab` to mount `/home` and `/usr` from the
+server. In `/diskless/n1/etc/fstab`:
+
+ # /etc/fstab: static file system information.
+ #
+ # <file system> <mount point> <type> <options> <dump> <pass>
+ # automatically mount nfs root and proc through other means
+ 192.168.2.100:/home /home nfs defaults,nolock 0 0
+ 192.168.2.100:/usr /usr nfs defaults,nolock 0 0
+ # we're diskess so we don't need to mount the hard disk sda :)
+ #/dev/sda1 / ext3 defaults,errors=remount-ro 0 1
+ /dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0
+ /dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
+
+
+<a name="kernel" />
+
+### Kernel and initial ramdisk
+
+The kernel version number shows up often in this section. You can
+determine your kernel version number (in my case 2.6.18-6-686) with
+`uname -r`. Because kernel versions change fairly frequently, I'll
+use `KERNEL_VERSION` to denote the kernel version string.
+
+Your kernel must be compiled with NFS root support if it's going to have an [NFS root][nfsroot].
+You can determine whether your kernel supports NFS roots with
+
+ # grep 'ROOT_NFS' /diskless/n1/boot/config-KERNEL-VERSION
+
+I didn't have it in my default debian etch 2.6.18-6-686 kernel, so I
+had to recompile my kernel (see the [Kernel Appendix][app.kernel] and
+[Falko's notes][kernel]). My compiled kernel had a version string
+`2.6.18-custom`.
+
+Most kernels boot using an [initial ramdisk][initrd] (a compressed
+root filesytem that lives in RAM). This ramdisk contains the
+necessary programs and scripts for booting the kernel. We need to
+create a ramdisk that can handle an NFS root, so `chroot` into your
+client filesystem and install some tools
+
+ chroot# apt-get install initramfs-tools
+
+Configure future ramdisks for NFS mounting with
+`/etc/initramfs-tools/initramfs.conf`:
+
+ # Configuration file for mkinitramfs(8). See initramfs.conf(5).
+ …
+ BOOT=nfs # was BOOT=local
+ …
+
+Compile a new `initrd` with
+
+ chroot# update-initramfs -u
+
+If you compiled your own kernel as in [Kernel Appendix][app.kernel]
+after setting up `initramfs.conf`, an appropriate ramdisk should have
+been created automatically.
+
+You can examine the contents of your ramdisk with
+
+ $ cp /diskless/n1/boot/initrd.img-2.6.18-6-686 initrd.img.gz
+ $ gunzip initrd.img.gz
+ $ mkdir initrd
+ $ cd initrd/
+ $ cpio -i --make-directories < ../initrd.img
+
+
+<a name="synthesis" />
+
+Synthesis
+---------
+
+To configure PXE, we need to bring `pxelinux.0` into our new
+`tftpboot` directory
+
+ # cp /usr/lib/syslinux/pxelinux.0 /diskless/tftpboot/
+
+We also need to bring in our kernel image and initial ramdisk
+
+ # cd /diskless/tftpboot
+ # ln -s /diskless/n1/boot/initrd.img-2.6.18-custom
+ # ln -s /diskless/n1/boot/vmlinuz-2.6.18-custom
+
+`atftpd` handles the symbolic links, but if your TFTP server doesn't,
+you'll have to copy the image and ramdisk over instead.
+
+At this point you should test your TFTP server with test transfers.
+Install the atftp client
+
+ # apt-get install atftp
+
+And attempt to transfer the important files.
+
+ $ atftp 192.168.2.100
+ tftp> status
+ Connected: 192.168.2.100 port 69
+ Mode: octet
+ Verbose: off
+ Trace: off
+ Options
+ tsize: disabled
+ blksize: disabled
+ timeout: disabled
+ multicast: disabled
+ mtftp variables
+ client-port: 76
+ mcast-ip: 0.0.0.0
+ listen-delay: 2
+ timeout-delay: 2
+ Last command: quit
+ tftp> get pxelinux.0
+ tftp> get initrd.img-2.6.18-custom
+ tftp> get vmlinuz-2.6.18-custom
+ tftp> quit
+ $ ls -l
+ …
+ -rw-r--r-- 1 sysadmin sysadmin 4297523 2008-05-30 09:27 initrd.img-2.6.18-custom
+ -rw-r--r-- 1 sysadmin sysadmin 13480 2008-05-30 09:26 pxelinux.0
+ -rw-r--r-- 1 sysadmin sysadmin 1423661 2008-05-30 09:27 vmlinuz-2.6.18-custom
+ …
+
+If this doesn't work, look for errors in `/var/log/syslog` and
+`/var/log/atftpd.log` and double check your typing in the `atftpd`
+configuration file.
+
+The last stage is to configure the `pxelinux.0` bootloader. Create a
+configuration directory in `tftboot` with
+
+ # mkdir /diskless/tftpboot/pxelinux.cfg
+
+When each client loads `pxelinux.0` during the boot, they look for a
+configuration file in `pxelinux.cfg`. The loader runs through a
+sequence of possible config file names, as described in
+`pxelinux.doc`. We'll have different rood directories for each of our
+nodes, so we need a seperate config for each of them. In order to
+make our configs machine-specific, we'll use the ethernet (MAC)
+address file-name scheme. That is, for a machine with MAC address
+AA:BB:CC:DD:EE:FF, we make the file
+`pxelinux.cgf/01-aa-bb-cc-dd-ee-ff`. TODO: base config on IP address.
+In `/diskless/tftpboot/pxelinux.cfg/01-aa-bb-cc-dd-ee-ff`:
+
+ default linux
+
+ label linux
+ kernel vmlinuz-2.6.18-custom
+ append root=/dev/nfs initrd=initrd.img-2.6.18-custom
+ nfsroot=192.168.2.100:/diskless/n1,tcp ip=dhcp rw
+
+Note that the `append`ed args should all be on a single, long line,
+since I haven't been able to discover if `pxelinux` recognizes escaped
+endlines yet. This file is basically like a `grub` or `lilo` config
+file, and you can get fancy with a whole menu, but since this is a
+cluster and not a computer lab, we don't need to worry about that.
+Note that this file was only for our first node (`n1`). You have to
+make copies for each of your nodes, with the appropriate file names
+and `nfsroot`s.
+
+The kernel options are fairly self explanatory except for the `tcp`
+for the `nfsroot` option, which says the client should mount the root
+directory using TCP based NFS. Traditional NFS uses UDP, which is
+faster, but possibly less reliable for large files (like our kernel
+and initrd). However I'm having trouble tracking down a reliable
+source for this. For now, consider the `tcp` a voodoo incantation to
+be attempted if the NFS booting isn't working.
+
+You're done! Plug a monitor into one of the clients and power her up.
+Everything should boot smoothly off the server, without touching the
+client's harddrive.
+
+
+Adding clients
+==============
+
+To add a new client node `nX` to the cluster, we need to do the
+following (which can be combined into an `add-client` script). First,
+we need to create a root directory for the new client
+
+ # cd /diskless/
+ # cp -rp n1 nX
+
+Now we need to export that directory
+
+ # echo '/diskless/nX 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)' >> /etc/exports
+ # exportfs -av
+
+Finally, we need to set up the booting and DHCP options
+
+ # cd /diskless/tftpboot
+ # sed 's/\/diskless\/n1/\/diskless\/nX/' 01-xx-xx-xx-xx-xx-xx > 01-yy-yy-yy-yy-yy-yy
+ # echo ' host n8 {
+ hardware ethernet YY:YY:YY:YY:YY:YY;
+ fixed-address 192.168.2.10X;
+ option root-path "192.168.2.100:/diskless/nX/";
+ option host-name "nX";
+ }' >> /etc/dhcpd.conf
+ # /etc/init.d/dhcp restart
+
+
+Appendix
+========
+
+<a name="app.kernel" />
+
+Compiling a kernel
+------------------
+
+See [Falko's notes][kernel] for an excellent introduction, and the
+[NFS-root mini-HOWTO][nfsroot-mini] for NSF root particulars.
+
+First, grab a bunch of useful compilation tools
+
+ chroot# apt-get install wget bzip2 kernel-package
+ chroot# apt-get install libncurses5-dev fakeroot build-essential initramfs-tools
+
+Some of these (e.g. `wget`) should already be installed, but apt-get
+will realize this, so don't worry about it. Configure `initramfs` for
+building NFS root-capable initial ramdisks by setting up
+`/etc/initramfs-tools/initramfs.conf` as explained in the [Kernel
+Section][sec.kernel]. For NSF root, your kernel needs the following
+options:
+
+ `IP_PNP_DHCP`
+ Networking
+ → Networking support (`NET [=y]`)
+ → Networking options
+ → TCP/IP networking (`INET [=y]`)
+ → IP: kernel level autoconfiguration (`IP_PNP =y`)
+ `ROOT_NFS` (`NET && NFS_FS=y && IP_PNP`)
+ File systems
+ → Network File Systems
+
+I also used the build-in NFS client instead of the module.
+Here is a `diff` of the original debian etch conf vs. mine:
+
+ $ diff /diskless/n1/boot/config-2.6.18-6-686 .config
+ 4c4
+ < # Sun Feb 10 22:04:18 2008
+ ---
+ > # Thu May 29 23:59:47 2008
+ 402c402,405
+ < # CONFIG_IP_PNP is not set
+ ---
+ > CONFIG_IP_PNP=y
+ > CONFIG_IP_PNP_DHCP=y
+ > CONFIG_IP_PNP_BOOTP=y
+ > CONFIG_IP_PNP_RARP=y
+ 3314c3317
+ < CONFIG_NFS_FS=m
+ ---
+ > CONFIG_NFS_FS=y
+ 3325c3328,3329
+ < CONFIG_LOCKD=m
+ ---
+ > CONFIG_ROOT_NFS=y
+ > CONFIG_LOCKD=y
+ 3328c3332
+ < CONFIG_NFS_ACL_SUPPORT=m
+ ---
+ > CONFIG_NFS_ACL_SUPPORT=y
+ 3330,3332c3334,3336
+ < CONFIG_SUNRPC=m
+ < CONFIG_SUNRPC_GSS=m
+ < CONFIG_RPCSEC_GSS_KRB5=m
+ ---
+ > CONFIG_SUNRPC=y
+ > CONFIG_SUNRPC_GSS=y
+ > CONFIG_RPCSEC_GSS_KRB5=y
+ 3485c3489
+ < CONFIG_CRYPTO_DES=m
+ ---
+ > CONFIG_CRYPTO_DES=y
+
+Compile your shiny, new kernel with
+
+ chroot# make-kpkg clean
+ chroot# fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
+
+The new kernel packages are in the `src` directory
+
+ chroot# cd /usr/src
+ chroot# ls -l
+
+Install the packages with
+
+ chroot# dpkg -i linux-image-2.6.18-custom_2.6.18-custom-10.00.Custom_i386.deb
+
+
+Troubleshooting
+---------------
+
+### No network devices available
+
+Getting
+
+ IP-Config: No network devices available.
+
+messages during the boot (*after* the kernel is successfully loaded!).
+According to [this post][no-net], the problem is due to a missing
+kernel driver.
+
+So I figured out what card I had:
+
+ # lspci
+ …
+ 03:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
+ 03:04.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
+ …
+
+The [ethernet HOWTO][ethernet] claimed that the `e1000` drivers were
+required for Intel gigabit cards, and indeed I had the e1000 module
+mounted on my server:
+
+ # lsmod | less
+ …
+ e1000 108480 0
+ …
+
+I reconfigured my kernel with (old vs new):
+
+ diff .config_mod_e1000 .config
+ 3,4c3,4
+ < # Linux kernel version: 2.6.18-custom
+ < # Fri May 30 00:13:47 2008
+ ---
+ > # Linux kernel version: 2.6.18
+ > # Fri May 30 22:21:29 2008
+ 1542c1542
+ < CONFIG_E1000=m
+ ---
+ > CONFIG_E1000=y
+
+After which I recompiled and reinstalled the kernel as in the [Kernel
+Appendix][app.kernel].
+
+### Waiting for `/usr/`
+
+On booting a client, I noticed a `Waiting for /usr/: FAILED` message
+just before entering runlevel 2. I attribute the error to a faulty
+boot order on the client not mounting it's fstab filesystems before
+trying to run something in `/usr/`. There don't seem to be any
+serious side effects though, since the wait times out, and by the time
+I can log in to the node, `/usr/` is mounted as it should be.
+
+
+[microwulf]: http://www.calvin.edu/~adams/research/microwulf/sys/microwulf_notes.pdf
+[debian-nfsroot]: http://www.jukie.net/~bart/blog/nfsroot-on-debian
+[kernel]: http://www.howtoforge.com/kernel_compilation_debian_etch
+[install]: http://www.debian.org/releases/stable/installmanual
+[netboot.tar.gz]: http://http.us.debian.org/debian/dists/etch/main/installer-i386/current/images/netboot/netboot.tar.gz
+[partition]: http://www.tldp.org/HOWTO/text/Partition
+[tasksel]: http://www.debian.org/releases/stable/i386/apds03.html.en
+[net-howto]: http://www.faqs.org/docs/Linux-HOWTO/Net-HOWTO.html
+[pxe]: http://syslinux.zytor.com/pxe.php
+[internals]: http://d-i.alioth.debian.org/doc/internals/
+[nfsroot]: http://www.kernel.org/doc/Documentation/filesystems/nfsroot.txt
+[initrd]: http://www.ibm.com/developerworks/linux/library/l-initrd.html
+[nfsroot-mini]: http://www.tldp.org/HOWTO/text/NFS-Root
+[no-net]: http://www.linuxquestions.org/questions/linux-networking-3/ip-config-no-network-device-available-591273/
+[ethernet]: http://tldp.org/HOWTO/Ethernet-HOWTO-4.html#ss4.24
+
+[sec.kernel]: #kernel
+[sec.synthesis]: #synthesis
+[app.kernel]: #app.kernel
+++ /dev/null
-\documentclass{article}
-
-\usepackage{url}
-\usepackage[colorlinks]{hyperref}
-\usepackage{fancyvrb}
-\usepackage{fullpage}
-
-%\renewcommand{\c}[1]{{\tt #1}}
-%\newcommand{\url}[1]{{\tt #1}}
-%\newenvironment{code}{\begin{verbatim}}{\end{verbatim}}
-\DefineVerbatimEnvironment{code}{Verbatim}{}
-\DefineVerbatimEnvironment{file}{Verbatim}{frame=single}
-% \begin{file}[label=/etc/example/file]
-% # some _ crazy \ contents
-% \end{file}
-\DefineVerbatimEnvironment{shell}{Verbatim}{}
-\newcommand{\inputfile}[1]{\VerbatimInput[frame=single,label=#1]{#1}}
-
-\newcommand{\Con}{\DefineShortVerb{\|}}
-\newcommand{\Coff}{\UndefineShortVerb{\|}}
-
-\author{W.\ Trevor King\thanks{Drexel University}}
-\date{\today}
-\title{Diskless Debian Etch}
-
-\begin{document}
-\maketitle
-\tableofcontents
-\Con
-
-\section{Overview}
-
-This HOWTO details the procedure I used to set up the abax cluster for NFS-rooted network booting.
-The system is useful in this case because it centralizes the installation in the head node (server), which makes maintaining, upgrading, or altering the computational nodes (clients) easier.
-
-This procedure follows mainly Tim Brom's Microwulf configuration notes\cite{microwulf} with two major differences.
-
-\begin{itemize}
- \item Microwulf uses Ubuntu (gutsy?), and I'm using Debian etch.
- \item Microwulf has a seperate partition for each client's root, populated with an independent installation from CD.
- I'm using a single partition for all of my clients, with the base system created using debootstrap (no CD).
-\end{itemize}
-
-For guidance in my deviations, I'm indebted to Bart Trojanowski's pxeboot and nfsroot notes\cite{debian-nfsroot} and Falko Timme's notes on kernel compilation in Debian Etch\cite{kernel}.
-
-
-\subsection{Physical setup}
-
-Our cluster has one server with eight clients.
-The server has two network cards, |eth0| and |eth1|.
-|eth1| is connected to the outsize world (WAN).
-All of the clients have one network card, |eth0|.
-All of the |eth0|s are connected together through a gigabit switch (LAN).
-
-
-\subsection{Notation}
-
-Throughout this HOWTO, I will use |#| as the prompt for root, |$| as the prompt for an unpriveledged user, and |chroot#| as the prompt for a root in a |chroot|ed environment.
-File contents will be framed with the full path listed labeling the frame
-\begin{file}[label=/path/to/file]
-Contents of file
-\end{file}
-All files are complete with the exception of lines containing |...|, in which case the meaning of the example should be clear from the context.
-
-
-\section{Basic server setup}
-
-\subsection{Installing the OS}
-
-Boot the server with the Debian installation kernel following one of the options in the Debian installation guide\cite{install}.
-I netbooted my server from one of the client nodes following this procedure to set up the DHCP and TFTP servers on the client and untarring \url{http://http.us.debian.org/debian/dists/etch/main/installer-i386/current/images/netboot/netboot.tar.gz} in my |tftpboot| directory.
-After netbooting from a client, don't forget to take that client down so you woln't have DHCP conflicts once you set up a DHCP server on your server.
-
-Install Debian in whatever manner seems most appropriate to you. I partitioned my 160 GB drive manually according to
-\begin{center}
-\begin{tabular}{l l l l}
- Mount point & Type & Size \\
- |/| & ext3 & 280 MB \\
- |/usr| & ext3 & 20 GB \\
- |/var| & ext3 & 20 GB \\
- |/swap| & swap & 1 GB \\
- |/tmp| & ext3 & 5 GB \\
- |/diskless| & ext3 & 20 GB \\
- |/home| & ext3 & 93.7 GB
-\end{tabular}
-\end{center}
-I went with a highly partitioned drive to ease mounting, since I will be sharing some partitions with my clients.
-To understand why partitioning is useful, see the Partition HOWTO\cite{partition}.
-
-You can install whichever packages you like, but I went with just the standard set (no Desktop, Server, etc.).
-You can adjust your installation later with any of (not an exhaustive list)
-\begin{itemize}
- \item |tasksel| : command line, coarse-grained package control\cite{tasksel}.
- \item |apt-get| : command line, fine-grained package control.
- \item |aptitude| : curses frontend for |apt-get|.
- \item |synaptic| : gtk+ frontend for |apt-get|.
- \item |dpkg| : command line, package-management without dependency checking.
-\end{itemize}
-
-The base install is pretty bare, but I don't need a full blown desktop, so I flesh out my system with
-\begin{shell}
-# apt-get install xserver-xorg fluxbox fluxconf iceweasel xterm xpdf
-# apt-get install build-essentials emacs21-nox
-\end{shell}
-which gives me a bare-bones graphical system (fire it up with |startx|), and
-a bunch of critical build tools (|make|, |gcc|, etc.).
-
-
-\subsection{Configuring networking}
-
-We need to set up our server so that |eth1| assumes it's appropriate static IP on the WAN, and |eth0| assumes it's appropriate static IP on the LAN.
-We achieve this by changing the default |/etc/network/interfaces| to
-\begin{file}[label=/etc/network/interfaces]
-# This file describes the network interfaces available on your system
-# and how to activate them. For more information, see interfaces(5).
-
-# The loopback network interface
-auto lo
-iface lo inet loopback
-
-allow-hotplug eth0 # start on boot & when plugged in
-iface eth0 inet static # static LAN interface
- address 192.168.2.100
- netmask 255.255.255.0
- broadcast 192.168.2.255
-
-allow-hotplug eth1 # start on boot & when plugged in
-#iface eth1 inet dhcp # WAN DHCP interface (not used)
-iface eth1 inet static # WAN static interface
- address XXX.XXX.YYY.YYY
- netmask 255.255.128.0
- broadcast XXX.XXX.127.255
- gateway XXX.XXX.ZZZ.ZZZ
-\end{file}
-where I've censored our external IPs for privacy.
-The netmask selects which addresses belong to which networks.
-The way we've set it up, all 192.168.2.xxx messages will be routed out |eth0|, and everything else will go through |eth1| to it's gateway.
-See the Net-HOWTO\cite{net-howto} for more details.
-
-
-\section{Remote booting}
-
-\subsection{Server services}
-
-The clients will boot remotely using the Pre eXecution Environment (PXE).
-The boot procedure is
-\begin{enumerate}
- \item Client powers on.
- \item Client BIOS comes up, detects attached devices, and looks for a DHCP server for advice on network booting.
- \item DHCP server gives client an IP address, domain name, host name, the IP address of the TFTP server, and the location of the bootloader on the TFTP server.
- \item Client gets bootloader from TFTP server.
- \item BIOS hands over control to bootloader.
- \item Bootloader gets kernel and initial ramdisk from TFTP server.
- \item Bootloader hands over control to kernel
- \item Kernel starts up the system, mounting root via NFS.
- \item \ldots after this point, it's just like a normal boot process.
-\end{enumerate}
-We can see that we need to set up DHCP, TFTP, and NFS servers (not necessarily on the same server, but they are in our case).
-
-\subsubsection{pxelinux}
-
-The bootloader can be obtained with
-\begin{shell}
-# apt-get install syslinux
-\end{shell}
-which installs it to |/usr/lib/syslinux/pxelinux.0| along with a manual\cite{pxe} and some other |syslinux| tools.
-
-\subsubsection{DHCP}
-
-Install a server with
-\begin{shell}
-# apt-get install dhcp
-\end{shell}
-
-Configure the server with
-\begin{file}[label=/etc/dhcpd.conf]
-allow bootp; # maybe?
-allow booting;# maybe?
-option domain-name "your.domain.com";
-option domain-name-servers XXX.XXX.XXX.XXX,YYY.YYY.YYY.YYY;
-
-subnet 192.168.2.0 netmask 255.255.255.0 {
- range 192.168.2.150 192.168.2.200; # non-static IP range
- option broadcast-address 192.168.2.255;
- option routers 192.168.2.100; # Gateway server
- next-server 192.168.2.100; # TFTP server
- filename "pxelinux.0"; # bootloader
-
- host n1 {
- hardware ethernet ZZ:ZZ:ZZ:ZZ:ZZ:ZZ;
- fixed-address 192.168.2.101;
- option root-path "192.168.2.100:/diskless/n1";
- option host-name "n1";
- }
- ... more hosts for other client nodes ...
-}
-\end{file}
-This assigns the client a static hostname, domain name, and IP address according to it's ethernet address (aka MAC address).
-It also tells all the clients to ask the TFTP server on 192.168.2.100 for the bootloader |pxelinux.0|.
-For extra fun, it tells the clients to send packets to the router at 192.168.2.100 if they can't figure out where they should go, and to use particular DNS servers to resolve domain names to IP addresses.
-This gives them access to the outside WAN.
-I don't know yet if the booting options are necessary, since I don't know what they do.
-
-We also need to ensure that the DHCP server only binds to |eth0|, since starting a DHCP server on your WAN will make you unpopular with your ISP.
-\begin{file}[label=/etc/default/dhcp]
-INTERFACES="eth0"
-\end{file}
-
-Once the DHCP server is configured, you can start it with
-\begin{shell}
-# /etc/init.d/dhcp restart
-\end{shell}
-
-Check that the server is actually up with
-\begin{shell}
-# ps -e | grep dhcp
-\end{shell}
-and if it is not, look for error messages in
-\begin{shell}
-# grep -i dhcp /var/log/syslog
-\end{shell}
-
-
-\subsubsection{TFTP}
-
-There are several TFTP server packages.
-We use |atftpd| here, but |tftp-hpa| is also popular.
-Install |atftpd| with
-\begin{shell}
-# apt-get install atftpd xinetd
-\end{shell}
-where |xinetd| is a super-server (replacing |inetd|, see |man xinetd| for details).
-Configure |atftpd| with
-\begin{file}[label=/etc/xinetd.d/atftpd]
-service tftp
-{
- disable = no
- socket_type = dgram
- protocol = udp
- wait = yes
- user = nobody
- server = /usr/sbin/in.tftpd
- server_args = --tftpd-timeout 300 --retry-timeout 5 --bind-address 192.168.2.100
- --mcast-port 1758 --mcast-addr 239.239.239.0-255 --mcast-ttl 1
- --maxthread 100 --logfile /var/log/atftpd.log --verbose=10
- /diskless/tftpboot
-}
-\end{file}
-Note that the |server_args| should all be on a single, long line, since I haven't been able to discover if |xinetd| recognizes escaped endlines yet.
-This configuration tells |xinetd| to provide TFTP services by running |in.tftpd| (the daemon form of |atftpd|) as user |nobody|.
-Most of the options we pass to |in.tftpd| involve multicasting, which I believe is only used for MTFTP (which |pxelinux.0| doesn't use).
-|--logfile /var/log/atftpd.log --verbose=10| logs lots of detail to |/var/log/atftpd.log| if it exists.
-You can create it with
-\begin{shell}
-# touch /var/log/atftpd.log
-# chown nobody.nogroup /var/log/atftpd.log
-\end{shell}
-
-The most important argument is |/diskless/tftpboot|, which specifies the root of the TFTP-served filesystem (feel free to pick another location if you would like).
-This is where we'll put all the files that the TFTP will be serving.
-It needs to be read/writable by |nobody|, so create it with
-\begin{shell}
-# mkdir \tftpboot
-# chmod 777 tftpboot
-\end{shell}
-(TODO: possibly set the sticky bit, remove writable?)
-
-Finally, we need to restart the |xinetd| server so it notices the new |atftpd| server.
-\begin{shell}
-# /etc/init.d/xinetd restart
-\end{shell}
-
-Check that the |xinetd| server is up with
-\begin{shell}
-# ps -e | grep xinetd
-\end{shell}
-and look for error messages in either of
-\begin{shell}
-# grep -i dhcp /var/log/syslog
-\end{shell}
-Just having |xinetd| up cleanly doesn't prove that |atftpd| is working though, it just shows that the |atftpd| configuration file wasn't too bungled.
-To actually test |atftpd| we need to wait to Section \ref{sec.synthesis} when we actually have files to test-transfer.
-
-
-\subsubsection{NFS}
-
-Install the NFS utilities on the server with
-\begin{shell}
-# apt-get install nfs-common nfs-kernel-server
-\end{shell}
-We go with the kernel server because we want fast NFS, since we'll be doing a lot of it.
-Set the NFS server up to export the root file systems and the user's home directories with
-\begin{file}[label=/etc/exports]
-/diskless/n1 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
-... other node root exports ...
-/diskless 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check) # unnecessary
-/home 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
-/usr 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)
-\end{file}
-Then let the NFS server know we've changed the |exports| file with
-\begin{shell}
-# exportfs -av # TODO: -r?
-\end{shell}
-
-Test that the NFS server is working properly by |ssh|ing onto one of the clients and running
-\begin{shell}
-client# mkdir /mnt/n1
-client# mount 192.168.2.100:/diskless/n1 /mnt/n1
-client# ls /mnt/n1
-... some resonable contents ...
-client# umount /mnt/n1
-client# rmdir /mnt/n1
-\end{shell}
-
-
-\subsection{Client setup}
-
-The only client setup that actually happens on the client is changing the BIOS boot order to preferentially boot from the network.
-Consult your motherboard manual for how to accomplish this.
-It should be simple once you get into the BIOS menu, which you generally do by pressing |del|, |F2|, |F12|, or some such early in your boot process.
-\emph{Everything else happens on the server}.
-
-
-\subsubsection{Root file system}
-
-We want to install a basic Debian setup on our clients.
-Since each client doesn't have it's own, private partition, we need to install Debian using |debootstrap|.
-\begin{shell}
-# apt-get install debootstrap
-# mkdir /diskless/n1
-# debootstrap --verbose --resolve-deps etch /diskless/n1
-# chroot /diskless/n1
-chroot# tasksel install standard
-chroot# dpkg-reconfigure locales
-chroot# apt-get install kernel-image-2.6-686 openssh-server nfs-client
-\end{shell}
-TODO: what get's installed with standard?
-See |/usr/share/tasksel/debian-tasks.desc| for a list of possible tasks and \url{http://d-i.alioth.debian.org/doc/internals/} for details on how a full installation from CD or netboot.
-
-We can also add a few utilities so we can work in our |chroot|ed environment
-\begin{shell}
-chroot# apt-get install emacs21-nox
-\end{shell}
-
-
-\subsubsection{Configuring etc}
-
-The client will be getting its hostnames from the DHCP server, so remove the default
-\begin{shell}
-# rm /diskless/n1/etc/hostname
-\end{shell}
-
-We also need to setup the |fstab| to mount |/home| and |/usr| from the server
-\begin{file}[label=/diskless/n1/etc/fstab]
-# /etc/fstab: static file system information.
-#
-# <file system> <mount point> <type> <options> <dump> <pass>
-# automatically mount nfs root and proc through other means
-192.168.2.100:/home /home nfs defaults,nolock 0 0
-192.168.2.100:/usr /usr nfs defaults,nolock 0 0
-# we're diskess so we don't need to mount the hard disk sda :)
-#/dev/sda1 / ext3 defaults,errors=remount-ro 0 1
-/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0
-/dev/fd0 /media/floppy0 auto rw,user,noauto 0 0
-\end{file}
-
-
-\subsubsection{Kernel and initial ramdisk}
-\label{sec.kernel}
-
-The kernel version number shows up often in this section.
-You can determine your kernel version number (in my case 2.6.18-6-686) with |uname -r|.
-Because kernel versions change fairly frequently, I'll use |KERNEL_VERSION| to denote the kernel version string.
-
-Your kernel must be compiled with NFS root support if it's going to have an NFS root\cite{nfsroot}.
-You can determine whether your kernel supports NFS roots with
-\begin{shell}
-# grep 'ROOT_NFS' /diskless/n1/boot/config-KERNEL-VERSION
-\end{shell}
-I didn't have it in my default debian etch 2.6.18-6-686 kernel, so I had to recompile my kernel (see Appendix \ref{app.kernel} and Falko's notes\cite{kernel}).
-My compiled kernel had a version string |2.6.18-custom|.
-
-Most kernels boot using an initial ramdisk (a compressed root filesytem that lives in RAM\cite{initrd}.
-This ramdisk contains the necessary programs and scripts for booting the kernel.
-We need to create a ramdisk that can handle an NFS root, so |chroot| into your client filesystem and install some tools
-\begin{shell}
-chroot# apt-get install initramfs-tools
-\end{shell}
-Configure future ramdisks for NFS mounting with
-\begin{file}[label=/etc/initramfs-tools/initramfs.conf]
-# Configuration file for mkinitramfs(8). See initramfs.conf(5).
-...
-BOOT=nfs # was BOOT=local
-...
-\end{file}
-Compile a new |initrd| with
-\begin{shell}
-chroot# update-initramfs -u
-\end{shell}
-If you compiled your own kernel as in Appendix \ref{app.kernel} after setting up |initramfs.conf|, an appropriate ramdisk should have been created automatically.
-
-You can examine the contents of your ramdisk with
-\begin{shell}
-$ cp /diskless/n1/boot/initrd.img-2.6.18-6-686 initrd.img.gz
-$ gunzip initrd.img.gz
-$ mkdir initrd
-$ cd initrd/
-$ cpio -i --make-directories < ../initrd.img
-\end{shell}
-
-
-\subsection{Synthesis}
-\label{sec.synthesis}
-
-To configure PXE, we need to bring |pxelinux.0| into our new |tftpboot| directory
-\begin{shell}
-# cp /usr/lib/syslinux/pxelinux.0 /diskless/tftpboot/
-\end{shell}
-
-We also need to bring in our kernel image and initial ramdisk
-\begin{shell}
-# cd /diskless/tftpboot
-# ln -s /diskless/n1/boot/initrd.img-2.6.18-custom
-# ln -s /diskless/n1/boot/vmlinuz-2.6.18-custom
-\end{shell}
-|atftpd| handles the symbolic links, but if your TFTP server doesn't, you'll have to copy the image and ramdisk over instead.
-
-At this point you should test your TFTP server with test transfers.
-Install the atftp client
-\begin{shell}
-# apt-get install atftp
-\end{shell}
-And attempt to transfer the important files.
-\begin{shell}
-$ atftp 192.168.2.100
-tftp> status
-Connected: 192.168.2.100 port 69
-Mode: octet
-Verbose: off
-Trace: off
-Options
- tsize: disabled
- blksize: disabled
- timeout: disabled
- multicast: disabled
-mtftp variables
- client-port: 76
- mcast-ip: 0.0.0.0
- listen-delay: 2
- timeout-delay: 2
-Last command: quit
-tftp> get pxelinux.0
-tftp> get initrd.img-2.6.18-custom
-tftp> get vmlinuz-2.6.18-custom
-tftp> quit
-$ ls -l
-...
--rw-r--r-- 1 sysadmin sysadmin 4297523 2008-05-30 09:27 initrd.img-2.6.18-custom
--rw-r--r-- 1 sysadmin sysadmin 13480 2008-05-30 09:26 pxelinux.0
--rw-r--r-- 1 sysadmin sysadmin 1423661 2008-05-30 09:27 vmlinuz-2.6.18-custom
-...
-\end{shell}
-If this doesn't work, look for errors in |/var/log/syslog| and |/var/log/atftpd.log| and double check your typing in the |atftpd| configuration file.
-
-The last stage is to configure the |pxelinux.0| bootloader.
-Create a configuration directory in |tftboot| with
-\begin{shell}
-# mkdir /diskless/tftpboot/pxelinux.cfg
-\end{shell}
-When each client loads |pxelinux.0| during the boot, they look for a configuration file in |pxelinux.cfg|.
-The loader runs through a sequence of possible config file names, as described in |pxelinux.doc|.
-We'll have different rood directories for each of our nodes, so we need a seperate config for each of them.
-In order to make our configs machine-specific, we'll use the ethernet (MAC) address file-name scheme.
-That is, for a machine with MAC address AA:BB:CC:DD:EE:FF, we make the file |pxelinux.cgf/01-aa-bb-cc-dd-ee-ff|.
-TODO: base config on IP address.
-\begin{file}[label=/diskless/tftpboot/pxelinux.cfg/01-aa-bb-cc-dd-ee-ff]
-default linux
-
-label linux
- kernel vmlinuz-2.6.18-custom
- append root=/dev/nfs initrd=initrd.img-2.6.18-custom
- nfsroot=192.168.2.100:/diskless/n1,tcp ip=dhcp rw
-\end{file}
-Note that the |append|ed args should all be on a single, long line, since I haven't been able to discover if |pxelinux| recognizes escaped endlines yet.
-This file is basically like a |grub| or |lilo| config file, and you can get fancy with a whole menu, but since this is a cluster and not a computer lab, we don't need to worry about that.
-Note that this file was only for our first node (|n1|).
-You have to make copies for each of your nodes, with the appropriate file names and |nfsroot|s.
-
-The kernel options are fairly self explanatory except for the |tcp| for the |nfsroot| option, which says the client should mount the root directory using TCP based NFS.
-Traditional NFS uses UDP, which is faster, but possibly less reliable for large files (like our kernel and initrd).
-However I'm having trouble tracking down a reliable source for this.
-For now, consider the |tcp| a voodoo incantation to be attempted if the NFS booting isn't working.
-
-You're done!
-Plug a monitor into one of the clients and power her up.
-Everything should boot smoothly off the server, without touching the client's harddrive.
-
-
-\section{Adding clients}
-
-To add a new client node |nX| to the cluster, we need to do the following (which can be combined into an |add-client| script).
-First, we need to create a root directory for the new client
-\begin{shell}
-# cd /diskless/
-# cp -rp n1 nX
-\end{shell}
-Now we need to export that directory
-\begin{shell}
-# echo '/diskless/nX 192.168.2.0/24(rw,no_root_squash,sync,no_subtree_check)' \
- >> /etc/exports
-# exportfs -av
-\end{shell}
-Finally, we need to set up the booting and DHCP options
-\begin{shell}
-# cd /diskless/tftpboot
-# sed 's/\/diskless\/n1/\/diskless\/nX/' 01-xx-xx-xx-xx-xx-xx > 01-yy-yy-yy-yy-yy-yy
-# echo ' host n8 {
- hardware ethernet YY:YY:YY:YY:YY:YY;
- fixed-address 192.168.2.10X;
- option root-path "192.168.2.100:/diskless/nX/";
- option host-name "nX";
- }' >> /etc/dhcpd.conf
-# /etc/init.d/dhcp restart
-\end{shell}
-
-
-
-\phantomsection
-\addcontentsline{toc}{section}{Appendix}
-\appendix
-
-
-\section{Compiling a kernel}
-\label{app.kernel}
-
-See Falko's notes\cite{kernel} for and excellent introduction, and the NFS-root mini-HOWTO\cite{nfsroot-mini} for NSF root particulars.
-
-First, grab a bunch of useful compilation tools
-\begin{shell}
-chroot# apt-get install wget bzip2 kernel-package
-chroot# apt-get install libncurses5-dev fakeroot build-essential initramfs-tools
-\end{shell}
-Some of these (e.g.\ |wget|) should already be installed, but apt-get will realize this, so don't worry about it.
-Configure |initramfs| for building NFS root-capable initial ramdisks by setting up \\ %HACK, force newline
-|/etc/initramfs-tools/initramfs.conf| as explained in Section \ref{sec.kernel}.
-For NSF root, your kernel needs the following options\cite{nfsroot-mini}:\\
-\begin{tabular}{l l}
- |IP_PNP_DHCP| & Networking \\
- & $\quad\rightarrow$ Networking support (|NET [=y]|) \\
- & $\quad\quad\rightarrow$ Networking options \\
- & $\quad\quad\quad\rightarrow$ TCP/IP networking (|INET [=y]|) \\
- & $\quad\quad\quad\quad\rightarrow$ IP: kernel level autoconfiguration (|IP_PNP =y|) \\
- |ROOT_NFS|(|NET && NFS_FS=y && IP_PNP|) & File systems \\
- & $\quad\rightarrow$ Network File Systems
-\end{tabular}
-
-I also used the build-in NFS client instead of the module.
-Here is a |diff| of the original debian etch conf vs.\ mine:
-\begin{file}[label=diff /diskless/n1/boot/config-2.6.18-6-686 .config]
-4c4
-< # Sun Feb 10 22:04:18 2008
----
-> # Thu May 29 23:59:47 2008
-402c402,405
-< # CONFIG_IP_PNP is not set
----
-> CONFIG_IP_PNP=y
-> CONFIG_IP_PNP_DHCP=y
-> CONFIG_IP_PNP_BOOTP=y
-> CONFIG_IP_PNP_RARP=y
-3314c3317
-< CONFIG_NFS_FS=m
----
-> CONFIG_NFS_FS=y
-3325c3328,3329
-< CONFIG_LOCKD=m
----
-> CONFIG_ROOT_NFS=y
-> CONFIG_LOCKD=y
-3328c3332
-< CONFIG_NFS_ACL_SUPPORT=m
----
-> CONFIG_NFS_ACL_SUPPORT=y
-3330,3332c3334,3336
-< CONFIG_SUNRPC=m
-< CONFIG_SUNRPC_GSS=m
-< CONFIG_RPCSEC_GSS_KRB5=m
----
-> CONFIG_SUNRPC=y
-> CONFIG_SUNRPC_GSS=y
-> CONFIG_RPCSEC_GSS_KRB5=y
-3485c3489
-< CONFIG_CRYPTO_DES=m
----
-> CONFIG_CRYPTO_DES=y
-\end{file}
-
-Compile your shiny, new kernel with
-\begin{shell}
-chroot# make-kpkg clean
-chroot# fakeroot make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers
-\end{shell}
-The new kernel packages are in the src directory
-\begin{shell}
-chroot# cd /usr/src
-chroot# ls -l
-\end{shell}
-
-Install the packages with
-\begin{shell}
-chroot# dpkg -i linux-image-2.6.18-custom_2.6.18-custom-10.00.Custom_i386.deb
-\end{shell}
-
-\section{Troubleshooting}
-
-Getting
-\begin{shell}
-IP-Config: No network devices available.
-\end{shell}
-messages during the boot (\emph{after} the kernel is successfully loaded!).
-According to \url{http://www.linuxquestions.org/questions/linux-networking-3/ip-config-no-network-device-available-591273/},
-the problem is due to a missing kernel driver.
-
-So I figured out what card I had:
-\begin{shell}
-# lspci
-...
-03:03.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
-03:04.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet Controller
-...
-\end{shell}
-
-The ethernet HOWTO (\url{http://tldp.org/HOWTO/Ethernet-HOWTO-4.html#ss4.24}) claimed that the |e1000| drivers were required for Intel gigabit cards, and indeed I had the e1000 module mounted on my server:
-\begin{shell}
-# lsmod | less
-...
-e1000 108480 0
-...
-\end{shell}
-
-I reconfigured my kernel with (new vs. old):
-\begin{file}[label=diff .config .config\_mod\_e1000]
-3,4c3,4
-< # Linux kernel version: 2.6.18
-< # Fri May 30 22:21:29 2008
----
-> # Linux kernel version: 2.6.18-custom
-> # Fri May 30 00:13:47 2008
-1542c1542
-< CONFIG_E1000=y
----
-> CONFIG_E1000=m
-\end{file}
-
-After which I recompiled and reinstalled the kernel as in Appendix \ref{app.kernel}.
-
-
-\subsection{Waiting for /usr/}
-
-On booting a client, I noticed a |Waiting for /usr/: FAILED| message just
-before entering runlevel 2.
-I attribute the error to a faulty boot order on the client not mounting it's
-fstab filesystems before trying to run something in |/usr/|.
-There don't seem to be any serious side effects though, since the wait times
-out, and by the time I can log in to the node, |/usr/| is mounted as it should
-be.
-
-\begin{thebibliography}{99}
-\addcontentsline{toc}{section}{Bibliography}
-
-\bibitem{microwulf}
- Tim Brom.
- {\it Microwulf Software and Network Configuration Notes}.
- May 16, 2008.\\
- \url{http://www.calvin.edu/~adams/research/microwulf/sys/microwulf_notes.pdf}
-
-\bibitem{debian-nfsroot}
- Bart Trojanowski.
- {\it pxeboot and nfsroot with debian}.
- March 29, 2007.\\
- \url{http://www.jukie.net/~bart/blog/nfsroot-on-debian}
-
-\bibitem{kernel}
- Falko Timme.
- {\it How To Compile A Kernel - Debian Etch}.
- June 12, 2007.\\
- \url{http://www.howtoforge.com/kernel_compilation_debian_etch}
-
-\bibitem{install}
- Debian Installer team.
- {\it Debian GNU/Linux 4.0 -- Installation Guide}.\\
- \url{http://www.debian.org/releases/stable/installmanual}
-
-\bibitem{partition}
- Anthony Lissot et al.
- {\it Linux Partition HOWTO}.\\
- |/usr/share/doc/HOWTO/en-txt/Partition.gz|\\
- \url{http://www.tldp.org/HOWTO/text/Partition}
-
-\bibitem{tasksel}
- Debian installer appendix with |tasksel| description.
- {\it D.3. Installing Debian GNU/Linux from a Unix/Linux System} \\
- \url{http://www.debian.org/releases/stable/i386/apds03.html.en}
-
-\bibitem{net-howto}
- Joshua Drake.
- {\it Linux Networking HOWTO}. \\
- \url{http://www.faqs.org/docs/Linux-HOWTO/Net-HOWTO.html}
-
-\bibitem{pxe}
- H.\ Peter Anvin.
- {\it PXELINUX}.\\
- |/usr/share/doc/syslinux/pxelinux.doc.gz| \\
- \url{http://syslinux.zytor.com/pxe.php}
-
-\bibitem{nfsroot}
- Gero Kuhlmann et al.
- {\it Mounting the root filesystem via NFS (nfsroot)}.
- 2006.\\
- |/usr/share/doc/linux-doc-2.6.18/Documentation/nfsroot.txt.gz| in |linux-doc-2.6.18| package\\
- \url{http://www.kernel.org/doc/Documentation/filesystems/nfsroot.txt}.
-
-\bibitem{initrd}
- M.\ Tim Jones.
- {\it Linux initial RAM disk (initrd) overview}.
- IBM Linux DeveloperWorks:
- July 31, 2006.\\
- \url{http://www.ibm.com/developerworks/linux/library/l-initrd.html}
-
-\bibitem{nfsroot-mini}
- Andreas Kostyrka et al.
- {\it NFS-Root mini-HOWTO}.
- September 20, 2002.\\
- |/usr/share/doc/HOWTO/en-txt/NFS-Root.gz|\\
- \url{http://www.tldp.org/HOWTO/text/NFS-Root}
-
-
-\end{thebibliography}
-
-\end{document}
projects / blog.git / commitdiff