Working in a Ubuntu world now, I really miss the Anaconda installer that CentOS/Fedora used. Every time I find myself having a problem with the d-i installer when doing network PXE installs, I find myself grepping and tracing through the source code.
While doing a Ubuntu 18 network install with a preseed file (all of which had previously worked just fine, no changes) we hit this error after the installer brought up the network: Downloading a file failed: The installer failed to access the mirror. This may be a problem with your network, blah blah blah.
Getting on the machine and diving into syslog, this is where things were going south:
Oct 6 23:10:48 anna[4786]: 2020-10-06 23:10:48 \ URL:http://mirror.example.com/repository/ubuntu/dists/bionic/main/binary-amd64/Release \ [109/109] -> "-" [1] Oct 6 23:10:48 anna[4786]: WARNING **: bad d-i Packages file Oct 6 23:10:48 anna[4786]: 2020-10-06 23:10:48 URL:http://mirror.example.com/repository/ubuntu/dists/bionic/Release \ [22243/22243] -> "/tmp/_fetch-url_net-retriever-4838-Release.4839" [1] Oct 6 23:10:48 net-retriever: Not verifying Release signature: unauthenticated mode enabled Oct 6 23:12:34 anna[4786]: WARNING **: bad d-i Packages file
This was very confusing because watching the logs on the mirror server, Packages was never even attempted to be downloaded, so how could it complain about a bad Packages file? The Releases file seemed sane, downloading it with wget on another machine, size, checksum all lined up.
(Also for the love of god, why are there no pagers in the install environment? no less, no more, not even vim, which makes it agonizing to look at logs. There’s just nano.)
Further, going onto another machine at the same site, doing an apt clean and blowing away all of the apt cache directories, then doing an apt update, everything was fine. I was able to download the packages indexes and install a package just fine.
So what gives? The Internets didn’t have any good leads, everyone seems to hit this bad d-i Packages file error while installing from CD-ROMs in 2009.
I went directly to the anna source code to see exactly what it was doing to generate that error. It’s written in C and I couldn’t see any obvious things that were going wrong. I didn’t find any other way to generate some debug logs for what d-i was doing. The closest thing I could find was doing dconf debugging by putting DEBCONF_DEBUG=developer on the kernel command line and re-trying the network install. This didn’t give me any logging for what anna and other commands were doing, but I did notice around my error it was setting a retriever/net/error value.
Googling around for what might set retriever/net/error lead me to the source for /usr/lib/debian-installer/retriever/net-retriever.
(have I mentioned how much I hate all the janky shell scripts that this installer calls out to?)
Inside the net-retriever script it’s doing some things with Package files which caught my interest. On my trouble system, I shoved an echo $* >> /tmp/blah-cmds to get it to record all of the arguments that the script was getting and tell me its secrets.
I hit retry on the installer, got the same error, but got this output of things fed to net-retriever:
# cat /tmp/blah-cmds cleanup config packages /var/cache/anna/Packages packages_one mirror.example.com /repository/ubuntu /var/cache/anna/Packages bionic error packages packages /var/cache/anna/Packages packages_one mirror.example.com /repository/ubuntu /var/cache/anna/Packages bionic error packages cleanup
hmmm. I went back and added a set -x at the top of the file and ran the script with the packages_one line. (The installer environment uses Busybox, so can’t just say bash -x of course)
Looking at the trace output I saw things like this:
+ pkgfile=restricted/debian-installer/binary-amd64/Packages + grep restricted/debian-installer/binary-amd64/Packages$ /tmp/net-retriever-5295-Release + line= + [ 1 != 0 ] + continue + exit 1
What’s this debian-installer business?
We didn’t have that directory available on our repos, and it finally dawned on me that it’s looking through the Releases file for various debian-installer/.../Packages paths. Because our Releases file didn’t have entries for them, this is what was causing anna to throw the bad d-i Packages file error. My other system worked just fine because it wasn’t trying to fetch installer packages. blah!
My understanding of the problem was that in our Aptly mirrors we weren’t mirroring udeb for our various components. Once we fixed that, debian-installer became available and everything was happy.
