A/K/A Ubuntu for CentOS kickstart users
I needed to install Ubuntu on a server so I finally got around to figuring out how to do an automated, unattended installation. Specifically, how to install Ubuntu on bare-metal over the network using an existing PXE setup in an all-CentOS environment, as you would in a datacenter. I will attempt to compare CentOS+kickstart installations vs Ubuntu+preseed installations based on my experience. There are some initial similarities in how things are setup, but not too many.
This is not a how-to, but a bridge to get started with Ubuntu. I’ve linked to any scripts I’ve used and where I originally found them.
TL;DR: Overall I’m not impressed with Ubuntu’s way of doing things. While at the end of the day you get an OS on disk, it takes a lot more clever work if you want to add any sort of local customizations particularly around hardware. CentOS/kickstart installations provide much more flexibility to run more things before and during installation, and easier to get in and troubleshoot. Kickstart becomes very handy if you need to setup things (via script, not by hand) beforehand like hardware RAID, flash cards, or conditional partitioning of disks.
PXE / netboot setup
Setting up your PXE environment for Ubuntu is basically identical to handling CentOS kickstarts. You serve the Ubuntu kernel+netboot initrd via HTTP or TFTP, save the same sort of config file to your bootloader of choice (PXELINUX, iPXE, etc), and tweak some different kernel command line options, and that’s it.
Setting up a local release mirror (e.g. Xenial)
(Disclaimer: I’m serving Ubuntu releases and repositories from a CentOS system)
A local release mirror is similar to CentOS is that once you get the files on disk, they’re served up from your HTTP server to clients the same way. Creating an Ubuntu/apt mirror on a CentOS system takes some work because obviously there’s different package managers and layouts involved. You could rsync the entire upstream Ubuntu repo, but this seems to include a lot of non-used stuff. There’s an RPM (in EPEL) and perl script called debmirror that takes care of using dpkg/apt tools and copying the bits you want onto your local repo server.
However there’s a snag with debmirror out of the box that I found. On CentOS, the RPM lays down both the script and a configuration file in /etc/debmirror.conf. It defaults to wanting to download the i386 version of the sid distribution. The comments in the config about command line options overriding the config are complete lies. Arguments are *appended* to what’s in debmirror.conf, so you wind up getting more than what you want.
The problem here (if you’re not aware of that config), is that debmirror will happily mirror your requested release, but then it also tries to download and do GPG checks of the sid release. Because you don’t have sid, it errors on a GPG check failure. Of course the Internet is stupid and says “just disable GPG checks”, instead of actually changing /etc/debmirror.conf.
Setting up local APT packge repositories
This is a local repository where you keep all of your own/third-party DEB packages (e.g. Chef client), seperate from the upstream (e.g. Xenial) release files.
Again, once you have a repository created it’s identical to a local CentOS mirror and served up over HTTP. To generate the package metadata on CentOS, you’d run createrepo. To create Ubuntu package metadata, you’re left to your own devices. There’s a decent shell script[1] out there called reindex_stable.sh which takes care of using dpkg tools to create all of the package indexes and generate SHA checksums. I’ve put my version of this up on Github[2].
Once you have the reindex script working it’s basically the same as running createrepo for CentOS.
- http://troubleshootingrange.blogspot.com/2012/09/hosting-simple-apt-repository-on-centos.html
- https://github.com/bwann/ubuntu/blob/master/reindex_stable.sh
Installer and configuration
Ok, this is the meat of the installation process. “Preseed” files are Ubuntu’s equivalent to kickstart config files. They answer installer questions as if you were going through them on console. Like kickstart configs, they can be served up over HTTP so the installer can download it.
Preseed files vs kickstart
Technically Ubuntu does let you use kickstart files[1], but it doesn’t support all the directives and options. Under the hood it’s just running shell scripts[2] to transform kickstart directives to preseed directives. Rather than try to shoehorn my existing kickstart configs to meet Ubuntu’s need, I just decided to go native and create my own preseed files.
The internet is littered with copypasta preseed files, just like random modem init string collections from the 90s. I’m not convinced people know exactly what options they’re using and why. An example of a basic preseed I used up with is up on github[3].
Pros of preseed files:
- They’re pretty short and succinct, mainly because there’s not a lot of heavy lifting you can do with them.
- Every possible configuration option available and set during installation will be written to /var/log/installer/cdebconf/questions.dat on an installed system
- There’s at least an option to change how filesystems are mounted, via UUID, label, or device names. UUIDs suck, use labels.
Cons of preseed files:
- The closest equivalent to kickstart’s %pre/%pre-install/%post sections are preseed/early_command, partman/early_command, preseed/late_command. You can only have a single command directive in the preseed file, so if you have a bunch of commands to run (or god forbid a script), you have to end each line with a semicolon and backslash. The entire line is consumed as one line, ugh. Or, you’re going to have to use this to download a script holding your bigger scripts.
- preseed/late_command is the equivalent to “%post --nochroot“. If you want to run commands within the target install, you have to use the in-target wrapper as part of the late_command string.
- https://help.ubuntu.com/community/KickstartCompatibility
- http://bazaar.launchpad.net/~ubuntu-installer/kickseed/master/files/head:/handlers/
- https://github.com/bwann/ubuntu/blob/master/base-preseed16.cfg
SSH to an installing system
One of the big things I miss with Ubuntu’s installer is being able to SSH into a host while it’s being installed. On CentOS, you can pass the inst.sshd option and set a password with sshpw in a kickstart file. This lets you log in while Anaconda is doing its thing to look at logs, troubleshoot your pre/post scripts, attach to the tmux session Anaconda is using, or have your build system interrogate the system to track its process, among other things.
Ubuntu has “network-console” you can enable, but this only starts an interactive installer session when you ssh in. The menu will let you launch a shell and look at logs, but it’s no longer an unattended installation. I get it, I can see the use case, but it’s not what I want. There’s not an easy way to hack this into Ubuntu without some heroic preseed/early_command setup. It’s probably best to re-build your own installer image with OpenSSH, config, and a key baked into it. Otherwise your only option to check on an unattended installation progress is to hop on VGA or serial console.
Bugs!
I hit at least two installer bugs that forced me to change how I did things (as of November 2018):
- xfs root + UEFI booting: When using xfs for the root filesystem, the system dumped to a grub prompt on boot: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1652822
- GPG keychain for local/custom repo loaded too late: The installer tries to install packages from your local repo, but it hasn’t yet done an apt-key update, so trying to install from a local repo fails. https://bugs.launchpad.net/ubuntu/+source/base-installer/+bug/1553121
This about wraps up my experience with Ubuntu installs. I’ve installed exactly one system (many times) so I don’t know of any other tricks to throw at it.
