Place Ghantoos

Cocktail of PXE, Debian, preseed, IPMI & puppet

Tagged: , , ,

A couple of days ago, I had to quickly setup around 60 machines with a standard GNU/Debian install.
The first shot was done using Razor. Razor is a very neat piece of software, developped by puppetlabs, that lets you setup an “intelligent” PXE installation by booting a mini-linux kernel and getting information from the physical machine using facter; giving you the possibility to install/configure the machines depending on their disk configuration/CPU model/RAM size/Model/Serial number/etc. Unfortunately, this project depends on the ipxe project which did not support, at the time of the install, all the network cards that were used (Broadcom and Intel). So I decided to go with an old school PXE/preseed installation coupled with DNS, IPMI and puppet configuration.

The whole idea was to:
1- boot and install the machine
2- reboot on the installed OS
3- use the IPMI ip address of the machine in order to set the hostname
4- launch puppet to setup the machine

First, I had to actually install and setup the PXE environment. I will not be long on these steps as they can found on many blogs all over the Internet.

I used dnsmasq for the dhcp/tftp part, apt-cacher-ng to cache the apt downloads of the machines and limit bandwidth usage when insalling all the software and Apache2 for the preseed part.

apt-get install dnsmasq apt-cacher-ng apache2

Here are the configuration files of both applications:

~# vim /etc/dnsmasq.conf
# For debugging purposes, log each DNS query as it passes through
# dnsmasq.

# Log lots of extra information about DHCP transactions.

~# vim /etc/apt-cacher-ng/backends_debian

Then I setup the actual PXE boot structure.

mkdir /var/lib/tftpboot
cd /var/lib/tftpboot
mkdir -p /var/lib/tftpboot/debian/wheezy/amd64
cd /var/lib/tftpboot/debian/wheezy/amd64
mkdir -p /var/lib/tftpboot/debian/wheezy/amd64/boot-screens
cd /var/lib/tftpboot/debian/wheezy/amd64/boot-screens
mkdir -p /var/lib/tftpboot/pxelinux.cfg/
cd /var/lib/tftpboot/pxelinux.cfg/

As I said earlier, some machines had Broadcom network adapters, so the next step was to patch the initrd that is delivered by GNU/Debian in order to add the famous bnx2/bnx2x modules. Googling around I found a script that did the job for ISO images and initrd files. I hacked the script to add the wheezy distribution support and ran it.

sed -i 's/lenny/wheezy/' add-firmware-to
chmod +x add-firmware-to
./add-firmware-to initrd.gz initrd.nonfree.gz wheezy
mv initrd.gz
ln -s initrd.nonfree.gz initrd.gz
rm add-firmware-to

Now let’s write the pxelinux.cfg/default file.

vim /var/lib/tftpboot/pxelinux.cfg/default 
# D-I config version 2.0
include debian/wheezy/amd64/boot-screens/menu.cfg
default debian/wheezy/amd64/boot-screens/vesamenu.c32
prompt 1
timeout 3

DEFAULT wheezy_amd64

LABEL wheezy_amd64
        kernel debian/wheezy/amd64/linux
        append vga=normal initrd=debian/wheezy/amd64/initrd.gz auto=true interface=auto netcfg/dhcp_timeout=60 netcfg/choose_interface=auto priority=critical url= DEBCONF_DEBUG=5
        IPAPPEND 2

As all my machines have multiple network adapters (up to 8), I could not know on which interface number udev was going to allocate to the interface that was actually connected to the administration vlan.
On my first attempts the debian-installer only seemed to try to dhcp on one interface than returned an error telling that it was unable to setup the network correctly, and thus unable to continue with the network install. After some googling, I found the information on a forum linking to a launchpad bug, etc. The secret is to add “IPAPPEND 2” to the pxe boot sequence in order to pass the MAC address of the interface wich actually booted during the PXE phase to the debian-installer, so that it could DHCP on this particular interface, and not the others.

Ok, so at this point, all machines were able to boot and launch the installation, then try to request the preseed as specified in the default file. In order not to write a preseed from scratch, I based my preseed file on the one that is included in Razor, changing some puppet variables that did not apply anymore and the post-configuration.

~# mkdir /var/www/pxe
~# vim /var/www/pxe/preseed.cfg 
d-i debian-installer/locale string en_US
d-i console-keymaps-at/keymap select us
d-i console-setup/ask_detect boolean false
d-i keyboard-configuration/xkb-keymap select us
d-i keyboard-configuration/layoutcode string us
d-i netcfg/choose_interface select auto
d-i netcfg/get_hostname string noconfigured
d-i netcfg/get_domain string
d-i netcfg/no_default_route boolean true
d-i mirror/country string manual
d-i mirror/protocol string http
d-i mirror/http/hostname string
d-i mirror/http/directory string /debian/
d-i mirror/http/proxy string
d-i mirror/suite string wheezy
d-i time/zone string Europe/Paris
d-i clock-setup/utc boolean true
d-i clock-setup/ntp boolean true
d-i partman-auto/disk string /dev/sda
d-i partman-auto/method string lvm
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-md/device_remove_md boolean true
d-i partman-auto-lvm/guided_size string 200G
d-i partman-auto/choose_recipe select atomic
d-i partman/default_filesystem string ext4
d-i partman-lvm/confirm boolean true
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i partman-md/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select finish
d-i partman/confirm boolean true
d-i partman/confirm_nooverwrite boolean true
d-i passwd/root-login boolean true
d-i passwd/make-user boolean false
d-i passwd/root-password password password
d-i passwd/root-password-again password password
d-i user-setup/allow-password-weak boolean true
#d-i apt-setup/restricted boolean true
tasksel tasksel/first multiselect standard
d-i pkgsel/include string ruby openssh-server build-essential curl tcpdump facter puppet ethtool vim screen curl dmidecode lsb-release ntp ipmitool
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
popularity-contest popularity-contest/participate boolean false
d-i finish-install/reboot_in_progress note
# Once the installation is done we'll set the system up for some firstboot
# magic.
d-i preseed/late_command string chroot /target sh -c "/usr/bin/curl -o /tmp/postinstall && /bin/sh -x /tmp/postinstall"

As described in this preseed, it will install the operating system, then download the postinstall file, which will itself download the actual firstboot script, and create an init script that will only be launched at the first boot. This part was inspired by the great post by Brian Taylor.

~# vim /var/www/pxe/postinstall 

# grab our firstboot script
/usr/bin/curl -o /root/firstboot
chmod +x /root/firstboot

# create a service that will run our firstboot script
cat > /etc/init.d/firstboot << EOF
# Provides:        firstboot
# Required-Start:  $networking
# Required-Stop:   $networking
# Default-Start:   2 3 4 5
# Default-Stop:    0 1 6
# Short-Description: A script that runs once
# Description: A script that runs once

cd /root ; /usr/bin/nohup sh -x /root/firstboot &


# install the firstboot service
chmod +x /etc/init.d/firstboot
update-rc.d firstboot defaults

echo "finished postinst"
~# vim /var/www/pxe/firstboot


# This script will run the first time the system boots. Even
# though we've told it to run after networking is enabled,
# Introducing a brief sleep makes things work right all the
# time. The time for DHCP to catch up.
sleep 120

# install our new sources
cat > /etc/apt/sources.list <> /etc/modules

# get IPMI ip and detect hostname
ipmi_ip=`ipmitool lan print | grep "IP Address  " | sed 's/.*: //'`
hostname=`grep $ipmi_ip /tmp/ipmilist | awk '{print $1}'`
rm /tmp/ipmilist

# set hostname
echo $hostname > /etc/hostname

puppet agent --verbose --no-daemonize --certname=`cat /etc/hostname` --onetime --server

# Remove our firstboot service so that it won't run again
update-rc.d firstboot remove
rm /etc/init.d/firstboot /root/firstboot

# Reboot into the new kernel

In the firstboot script, I get the IPMI ip information using ipmitool(1), than deduct the hostname information from the ipmilist file. This could be directly done via a reverse DNS request; but I didn't have the time to setup this.

This allows the machine to connect to the puppetmaster server using its real hostname, and get itself configured (software + network).

The ipmilist file looks something like this:

I agree this is very old-school, not scalable, but I had to improvise to deliver these machine. Plus, it worked like a charm. :)

Ignace M

Tagged: , , ,

Leave a Reply