Verifying a Linux guide with no Linux machine

I had a NixOS installation guide to verify, and a machine incapable of running Linux. No Linux builder, no server at hand, a Mac.

The guide in question promised two mirrored SSDs and a boot that survives a dead disk. Taking it seriously meant answering questions like "can UEFI firmware read this partition" or "does this machine reboot with one disk missing", and none of those are settled by re-reading text.

Four tiers, then. And the interesting part is not that they cost more and more, it is that they do not answer the same question.

Four questions, not four degrees of realism

The temptation, when building a test bench, is to see a ladder of fidelity: the more it costs, the closer to real, and you stop when the budget runs out. Poor way to think about it. It suggests an expensive tier does everything a cheap one does, only better.

Each tier interrogates a different party:

No tier can answer on another’s behalf. Asking Linux whether firmware can read a partition is meaningless: Linux has the md driver, it will say yes, and it will be right about itself. That is the central flaw of the guide I was examining, and it is also the trap any single-storey test bench falls into.

Tier 1: evaluate without executing anything

The cheapest, and by far the best value. NixOS is a module system: hand it a configuration and read back what it thinks, without building or running anything.

nix-instantiate --eval --strict -E '
let n = import <nixpkgs/nixos> { configuration = ./configuration.nix; };
in {
  drv = n.config.system.build.toplevel.drvPath;
  warnings = n.config.warnings;
  failedAssertions = builtins.filter (a: !a.assertion) n.config.assertions;
}'

Three fields, three kinds of defect. drvPath forces full evaluation, so an unknown option or a wrong type blows up right here. warnings collects what modules flag without blocking, which is precisely what nobody reads in a scrolling terminal. assertions gives the incompatibilities nixpkgs took the trouble to forbid.

Seconds, on any machine where Nix evaluates, macOS included, since nothing is ever built for Linux.

On the guide under examination, this tier turned up an mdadm warning its author had not seen, two options renamed months ago, two packages that no longer exist under that name. Nothing spectacular, but it is what a reader meets on the first attempt.

One variant is worth the detour: rather than a copy of the configuration, I extract the block from the published Markdown and evaluate that text. Three lines of Python, and the whole class of errors where the article and the tested file have drifted apart disappears.

block = re.search(r'```nix\n(.*?)\n```', article[i:], re.S).group(1)

Tier 2: a Linux kernel without provisioning a virtual machine

For the block layer you need a real Linux kernel with md and btrfs. Not necessarily a virtual machine to install: if you have podman on a Mac, you already have a Linux machine running.

podman run --rm --privileged -v "$PWD/experiment.sh:/x.sh:z" alpine:latest sh /x.sh

Two things are missing inside that container, and both can be worked around.

There are no loop devices, so you create them. The loop major number is 7, and the minor is the number:

for i in 0 1 2 3; do [ -e /dev/loop$i ] || mknod /dev/loop$i b 7 $i; done

There is no udev either, so nobody creates the partition nodes: after a parted, no /dev/loop0p1 appears. The way round is more elegant than starting a daemon: a loop device can map exactly a partition’s region.

losetup -o $((1024*1024)) --sizelimit $((511*1024*1024)) /dev/loop2 disk.raw

That /dev/loop2 is, byte for byte, what mdadm would see of an sda1. You can create an array on it, format it, mount it, measure where the superblock lands.

Two traps cost me time. The first: /nix/store is not shared with the podman machine, so anything coming from the store must first be copied under the working directory before being mounted into a container. The second: podman caches its images by name, not by platform, so an amd64 image pulled earlier is silently reused when you ask for arm64. You have to be explicit, or watch mdadm fail for no visible reason:

podman run --rm --privileged --platform linux/arm64 ...

This tier answered everything to do with the disk: the superblock’s position per metadata version, btrfs refusing to mount an incomplete array, btrfs device remove failing on a two-disk mirror, and the fact that a wipefs over the whole disk leaves a previous install’s md superblock behind.

Tier 3: put the question to the firmware

Then comes the question Linux cannot settle. UEFI firmware reads the EFI partition before any kernel exists, and it has no md driver. Finding out whether it manages requires real firmware.

QEMU ships two, already in the package: EDK2 for UEFI, SeaBIOS for legacy BIOS. You build a disk image, boot it with no operating system at all, and read the serial console.

qemu-system-x86_64 -machine q35 -m 512 -nodefaults -no-reboot \
  -drive if=pflash,format=raw,unit=0,readonly=on,file="$QEMU/share/qemu/edk2-x86_64-code.fd" \
  -drive if=pflash,format=raw,unit=1,file=vars.fd \
  -drive file=disk-a.raw,format=raw,if=virtio \
  -display none -serial file:serial.log -net none

The firmware announces what it does. If it finds a bootloader it writes starting Boot0001, otherwise failed to load ... Not Found and then drops into its shell, whose mapping table says exactly which filesystems it managed to recognise. That is a very direct witness, and an entirely uncharitable one.

One obstacle remained, and I thought it fatal: I needed an x86_64 EFI binary to put on the image, and I had no Linux builder.

That is where I was wrong, and it is the insight the whole bench rests on. Nix substitutes closures for any platform. Building for x86_64-linux on a Mac is impossible; downloading it ready-built from the cache is not.

nix build --no-link --print-out-paths 'nixpkgs#legacyPackages.x86_64-linux.systemd'

And above all, that binary never needs to be executed by me. It only has to be copied into a FAT partition. The emulated firmware is what will execute it, and that firmware is x86_64. The distinction between building, executing and merely copying had escaped me, and it changed everything.

Emulation still has its limits, and the thing to do is work around them rather than conclude too fast. qemu-user mistranslates md’s ioctls: under emulation, mdadm --create answers

failed to set array info: Not a tty

which has nothing to do with the real problem. The same happens to grub-install, which probes the array with a RAID_VERSION ioctl. I first declared BIOS mode untestable because of this. That was wrong: the chain splits in two. grub-mkimage only packs modules and touches no block device, so it runs fine under emulation; grub-bios-setup writes sectors but is happy with an ordinary directory and never probes the array. The claim I wanted to test stays intact, because it is the prefix embedded in the image that sends GRUB to read the array at runtime.

When a step fails under emulation, the useful question is not "how do I emulate better" but "which part of this command actually needs the hardware".

Tier 4: install it for real

The first three tiers had answered all my questions, and I nearly stopped there. I was wrong, and this is the tier that taught me the most.

I thought I needed an x86_64-linux machine. I did not: the Mac is aarch64, and an ARM64 guest runs on it with hardware acceleration, so at near-native speed. I needed an aarch64 installation ISO, and NixOS publishes one you can simply download rather than build.

curl -L -o nixos.iso \
  https://channels.nixos.org/nixos-26.05/latest-nixos-minimal-aarch64-linux.iso
qemu-system-aarch64 -machine virt,accel=hvf -cpu host -smp 4 -m 4096 ...

From there nothing is emulated in the loose sense: a real udev, a real systemd initrd, a real nixos-install, real reboots. And unplugging a disk is done by removing a line from the QEMU command line, which is markedly more convenient than a screwdriver.

The real cost, with figures: a 1.7 GB ISO, two sparse 16 GiB disks, 16 recorded machine boots, and 77 minutes between the download and the last test. A boot-plus-check cycle takes one to three minutes.

That is little for what it returns. This tier found three errors, and two of them were in my corrected version of the guide, not in the original. I was about to publish a one-line fix that fixes nothing.

Driving a serial console without losing your mind

An installation ISO is interactive, and there is no channel other than the console. I wrote a small expect-style driver over QEMU’s serial socket.

qemu-system-aarch64 ... -serial unix:serial.sock,server,nowait

Two traps are worth knowing, because together they cost me an hour and neither looks like what it is.

Never stop reading. QEMU writes to the serial line in blocking fashion. If the driver stops reading, even for a five-second sleep, the buffer fills and it is the guest that freezes, in the middle of an escape sequence. The symptom is disorienting: the log stops dead, its final bytes a terminal-title sequence cut in half.

b'[?2004h\r\r\n\x1b[1;31m[\x1b]0;root@n'

The prompt was about to be written, the shell was waiting, and only the following bytes were missing. So the socket has to be drained continuously, including between two keystrokes.

Never synchronise on a marker you typed. My first driver sent command ; echo DONE and then waited for DONE. It found it immediately, because the terminal echoes back the line you type, marker included, before the command even runs. The driver then ran a command ahead of itself and everything after desynchronised. The right thing to synchronise on is the shell’s own prompt, which appears only once the command has finished.

One last piece of advice, which removes a lot of typing: in user-mode networking the guest reaches the host at 10.0.2.2. A one-line HTTP server on the host is enough to deliver scripts to the guest, which then only needs curl ... | sh. You iterate on a local file instead of retyping two hundred lines into a serial console.

What each tier found

The distribution surprised me. The cheap tiers found the spectacular defect, the one that makes the article: a perfectly installed machine that cannot boot. The expensive tier found the quiet ones, the sort you publish without seeing because they look like common sense.

And that is exactly the reason to build the top storey. A fix that looks obvious, that evaluates without error and that everybody recommends can still fix nothing. There was only one way to find out.

What I take away

The tiers are not a ladder you climb until the budget runs out. They are four parties to question, and you go to the one who holds the answer.

So the question to ask of a test bench is not "is this realistic enough" but "does this measurement cross the boundary the problem lives on". A guide that verifies its install by asking Linux will never learn what firmware thinks. A bench that stops at evaluation will never learn whether the machine boots.

Which is what happened to me, in fact, at a boundary I had not seen. I nearly published that degraded in the mount options was enough to reboot on one disk. The option is correct. It simply arrives after the moment that decides everything, and only a real boot could tell me so.

To reproduce

• QEMU for the ARM64 virt machine and its firmware options
• The NixOS ISOs, aarch64 images included, to download rather than build
• losetup(8), for --offset and --sizelimit
• The NixOS manual, for nixos-install and nixos-generate-config