Two mirrored SSDs under NixOS

Two SSDs, everything mirrored, and a machine that keeps booting when one disk dies. That is the goal, and it is half reachable.

I started from a guide written by a language model, nearly two and a half thousand lines, tidier than most human tutorials. I measured it layer by layer, then installed it for real in a virtual machine. Its UEFI layout boots on no machine at all, and none of its checks will tell you.

Better to announce the half that does not work straight away: a machine with a disk missing does not reboot on its own, and no one-line option changes that. I believed it did, I had written that it did, the real install proved me wrong.

The trap: do not put the ESP in a RAID under UEFI

This is the decision that shapes everything, and it has to be made before the first command.

The appealing idea is to join the two EFI partitions into a software RAID 1. The kernel sees them as a single volume, writes are duplicated, and if one disk dies the other already holds everything. No more syncing /boot by hand.

The problem is one of ordering. When a UEFI machine starts, the firmware has to read the EFI partition before any kernel exists. It has no md driver. It opens the partition, reads its first sector, and expects to find the header of a FAT filesystem there.

But mdadm writes metadata of its own onto every member of the array, and mdadm --create with no --metadata picks version 1.2, the one that puts its superblock 4 KiB from the start. Two identical arrays, one variable:

eb 58 90 is the jump instruction that opens a FAT32 boot sector, 55 aa its closing signature. Under 1.2 there is neither: the md superblock holds the spot, and the data starts a mebibyte further along than where the firmware is looking.

Booted under EDK2, with a real bootloader at the fallback path \EFI\BOOT\BOOTX64.EFI, the result is unambiguous:

BdsDxe: failed to load Boot0001 "UEFI Misc Device" from PciRoot(0x0)/Pci(0x1,0x0): Not Found
BdsDxe: failed to load Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x2,0x0): Not Found
BdsDxe: loading Boot0003 "EFI Internal Shell"

That shell’s mapping table settles the argument: it lists BLK0: through BLK5:, meaning the disks and their GPT partitions, ESP included. There is no FS0: entry at all. The firmware saw the partitions perfectly well. It found a filesystem on none of them.

Switching to --metadata 1.0 moves the superblock to the end and makes the read possible, which I verified. I do not recommend it: the firmware can then write into the ESP without going through md, for its boot variables or an update, and silently desync the mirror. Reasoning, not measurement, but enough to prefer something else.

The right answer under NixOS is called mirroredBoots. Two independent ESPs, each with its own FAT32, and NixOS writes both on every nixos-rebuild boot. The firmware has nothing left to understand: it reads an ordinary FAT partition, on whichever disk it still has.

Each ESP is self-sufficient: NixOS copies the bootloader and the kernels into it, once per mirroredBoots entry. On the installed machine both held two kernels each.

Measured under EDK2, blank NVRAM:

That is the exact step at which the original guide’s layout fails.

BIOS mode is a different world, and RAID on /boot genuinely works there. The firmware loads GRUB’s core.img from a small dedicated partition, as raw sectors, with no filesystem involved. That core.img carries the mdraid1x and fat modules: GRUB assembles the array itself and reads /boot from it. Booted under SeaBIOS, with metadata 1.2, the serial log contains the markers placed in a grub.cfg that exists only on the array:

MARKER-GRUB-READ-GRUB-CFG-FROM-MD-ARRAY
MARKER-ROOT-IS-md/0

GRUB resolved its root to md/0 on its own, and did so with either disk alone too. So the original guide’s mistake has nothing to do with mdadm. It is about which layer does the reading.

Partitioning

Both disks are about to be erased. Identify them by stable id, not by /dev/sda: enumeration order can change between boots, and these names are going to end up written into your configuration.

ls -l /dev/disk/by-id/ | grep -v part
DISK_A=/dev/disk/by-id/nvme-SAMSUNG_XXXX_YYYY
DISK_B=/dev/disk/by-id/nvme-SAMSUNG_XXXX_ZZZZ

The cleanup takes two passes, and the order is not negotiable:

mdadm --stop --scan 2>/dev/null || true

# a previous install's md superblocks live INSIDE the partitions
for p in "$DISK_A"-part* "$DISK_B"-part*; do
  [ -e "$p" ] || continue
  mdadm --zero-superblock "$p" 2>/dev/null || true
  wipefs --all --force "$p"
done

wipefs --all --force "$DISK_A"
wipefs --all --force "$DISK_B"

I first condensed this into one wipefs per disk, reasoning that destroying the partition table was enough. It is not: an md superblock lives inside the old partition’s region, where a whole-disk wipe never looks for it.

after a whole-disk wipefs only:
   mdadm --examine finds a superblock: yes
after wiping the partitions, then the disk:
   mdadm --examine finds a superblock: no

The original guide did erase the superblocks partition by partition. On that point it was right and I was wrong to simplify.

It also proposed dding over the end of each disk for the backup GPT, and that step really is unnecessary: wipefs already erases all three signatures.

/dev/loop0: 8 bytes were erased at offset 0x00000200 (gpt)
/dev/loop0: 8 bytes were erased at offset 0x3ffffe00 (gpt)
/dev/loop0: 2 bytes were erased at offset 0x000001fe (PMBR)

The second line is the backup header, at the end of the disk. Without wipefs, a dd over the start alone leaves that copy intact and sgdisk regenerates the primary header from it.

Under UEFI, one ESP per disk and the rest for data:

for d in "$DISK_A" "$DISK_B"; do
  parted -s "$d" mklabel gpt
  parted -s "$d" mkpart ESP fat32 1MiB 512MiB
  parted -s "$d" set 1 esp on
  parted -s "$d" mkpart primary 512MiB 100%
done

mkfs.fat -F 32 -n ESP-A "$DISK_A-part1"
mkfs.fat -F 32 -n ESP-B "$DISK_B-part1"

DATA_A="$DISK_A-part2"
DATA_B="$DISK_B-part2"

The two ESPs get two distinct labels, deliberately: they are two separate filesystems, and NixOS has to be able to mount them separately.

Under BIOS, three partitions, one of which is a two mebibyte region GRUB uses internally and that is neither formatted nor mounted:

for d in "$DISK_A" "$DISK_B"; do
  parted -s "$d" mklabel gpt
  parted -s "$d" mkpart primary 1MiB 3MiB
  parted -s "$d" set 1 bios_grub on
  parted -s "$d" mkpart primary 3MiB 515MiB
  parted -s "$d" mkpart primary 515MiB 100%
done

mdadm --create /dev/md0 --level=1 --raid-devices=2 --name=boot \
  "$DISK_A-part2" "$DISK_B-part2"
mkfs.fat -F 32 -n BOOT /dev/md0

DATA_A="$DISK_A-part3"
DATA_B="$DISK_B-part3"

btrfs RAID 1 and the subvolumes

Identical in both modes, except the data partition changes number: the second under UEFI, the third under BIOS. Hence the DATA_A and DATA_B in the blocks above. Inattention costs dearly here: under BIOS, part2 is a member of the /boot RAID, and an mkfs.btrfs over it would destroy booting.

mkfs.btrfs -L nixos -d raid1 -m raid1 "$DATA_A" "$DATA_B"

mount /dev/disk/by-label/nixos /mnt
btrfs subvolume create /mnt/@
btrfs subvolume create /mnt/@home
btrfs subvolume create /mnt/@snapshots
umount /mnt

-d raid1 -m raid1 mirrors data and metadata. The subvolumes separate what gets backed up from what gets replaced: @ the root, @home the accounts, @snapshots the snapshots.

Mounting

mount -o subvol=@,compress=zstd,noatime /dev/disk/by-label/nixos /mnt
mkdir -p /mnt/{home,.snapshots}
mount -o subvol=@home,compress=zstd,noatime /dev/disk/by-label/nixos /mnt/home
mount -o subvol=@snapshots,compress=zstd,noatime /dev/disk/by-label/nixos /mnt/.snapshots

Then the boot partitions, depending on the mode. Under UEFI, both ESPs:

mkdir -p /mnt/boot /mnt/boot-fallback
mount -o fmask=0077,dmask=0077 /dev/disk/by-label/ESP-A /mnt/boot
mount -o fmask=0077,dmask=0077 /dev/disk/by-label/ESP-B /mnt/boot-fallback

Under BIOS, the array:

mkdir -p /mnt/boot
mount -o fmask=0077,dmask=0077 /dev/md0 /mnt/boot

fmask=0077 and dmask=0077 reserve reading for root. FAT32 carries no Unix permissions, so those masks are the only lever, and the partition holds the kernel and the initrd. NixOS also points systemd’s random seed at it, whatever the bootloader.

The configuration, without overwriting what was detected

nixos-generate-config --root /mnt

Two files, and the original guide’s largest trap.

hardware-configuration.nix does not only hold hardware: it holds your filesystem layout, read off whatever is mounted at that moment. Hence the order of the preceding steps. Here is what it wrote during my install:

  boot.initrd.availableKernelModules = [ "virtio_pci" "virtio_scsi" "sr_mod" ];
  fileSystems."/" =
    { device = "/dev/disk/by-uuid/05a364d3-6496-4908-9a0d-6f0e3dc84948";
      fsType = "btrfs";
      options = [ "subvol=@" ];
    };
  fileSystems."/boot" =
    { device = "/dev/disk/by-uuid/C3C1-28D9";
      fsType = "vfat";
      options = [ "fmask=0077" "dmask=0077" ];
    };

It names volumes by UUID: the labels were only a convenience for typing the mounts. And it kept subvol= and the FAT masks, but dropped compress=zstd and noatime.

I installed a machine exactly that way, adding nothing. findmnt, on the booted system:

/              rw,relatime,discard=async,space_cache=v2,subvolid=256,subvol=/@
/home          rw,relatime,discard=async,space_cache=v2,subvolid=257,subvol=/@home
/boot          rw,relatime,fmask=0077,dmask=0077,codepage=437,...

No compression, relatime instead of noatime. The FAT masks survived. Stop there and you install a system with no compression, and nothing will tell you.

The original guide replaced configuration.nix with a cat > ..., overwriting the imports = [ ./hardware-configuration.nix ] line written seconds earlier. The excerpt above shows the price: the virtualisation profile and the initrd’s virtio_* modules, and with them the root, unfindable inside a virtual machine. On its configurations I measured the same loss in three places: an initrd reduced to ahci nvme sd_mod md_mod raid1, microcode no longer updated, redistributable firmware no longer installed.

Keeping the import and restating the fileSystems does not work either. I walked straight into it on my first attempt:

error: The option `fileSystems."/".device' has conflicting definition values:
- In `/mnt/etc/nixos/configuration.nix': "/dev/disk/by-label/nixos"
- In `/mnt/etc/nixos/hardware-configuration.nix': "/dev/disk/by-uuid/05a364d3-..."

The guide claims that "the configuration.nix version will override them". It conflicts, and evaluation stops dead.

The way out rests on a type difference between two neighbouring options. device is a string: two definitions contradict each other. options is a list: two definitions merge. So you add what is missing, without ever restating a device:

  fileSystems."/".options = [ "compress=zstd" "noatime" ];
  fileSystems."/home".options = [ "compress=zstd" "noatime" ];
  fileSystems."/.snapshots".options = [ "compress=zstd" "noatime" ];

Which evaluates to the full list, the device staying the one detection wrote:

[ "x-initrd.mount" "compress=zstd" "noatime" "subvol=@" ]

configuration.nix therefore restates no filesystem at all. Here it is in full.

Under UEFI:

{ config, pkgs, lib, ... }:
{
  imports = [ ./hardware-configuration.nix ];

  boot.loader.grub = {
    enable = true;
    efiSupport = true;
    efiInstallAsRemovable = true;
    mirroredBoots = [
      { path = "/boot"; devices = [ "/dev/disk/by-id/nvme-SAMSUNG_XXXX_YYYY" ]; }
      { path = "/boot-fallback"; devices = [ "/dev/disk/by-id/nvme-SAMSUNG_XXXX_ZZZZ" ]; }
    ];
  };
  boot.loader.efi.canTouchEfiVariables = false;

  boot.supportedFilesystems = [ "btrfs" ];
  boot.initrd.supportedFilesystems = [ "btrfs" ];

  fileSystems."/".options = [ "compress=zstd" "noatime" ];
  fileSystems."/home".options = [ "compress=zstd" "noatime" ];
  fileSystems."/.snapshots".options = [ "compress=zstd" "noatime" ];

  nix.package = pkgs.lix;

  services.btrfs.autoScrub = {
    enable = true;
    interval = "weekly";
    fileSystems = [ "/" ];
  };
  services.smartd = {
    enable = true;
    notifications.wall.enable = true;
  };

  networking.hostName = "nixos";
  networking.networkmanager.enable = true;
  time.timeZone = "Europe/Paris";
  i18n.defaultLocale = "en_GB.UTF-8";

  users.users.someone = {
    isNormalUser = true;
    extraGroups = [ "wheel" "networkmanager" ];
  };

  environment.systemPackages = with pkgs; [
    vim git btrfs-progs compsize mdadm smartmontools
  ];

  services.openssh.enable = true;

  system.stateVersion = "26.05";
}

Three lines deserve a word.

efiInstallAsRemovable = true makes the bootloader be written to the fallback path firmware looks for when no NVRAM entry guides it. That flag decides it, not GRUB by default. Two installs onto a fresh ESP, the flag the only difference:

with --removable:   <ESP>/EFI/BOOT/BOOTAA64.EFI
without:            <ESP>/EFI/NixOS/grubaa64.efi

An NVRAM entry names a specific partition by its id and dies with the disk that carries it, so the second path is reachable only through that entry. The fallback path is found on its own on whatever disk is left, which is what the firmware did in the single-disk test.

In exchange, nixpkgs forbids it alongside EFI variables, plainly:

If you wish to to use boot.loader.grub.efiInstallAsRemovable,
then turn off boot.loader.efi.canTouchEfiVariables

nix.package = pkgs.lix installs Lix, 2.95.4 as I write. The guide promised it in its filename and used it nowhere.

system.stateVersion carries the release you install from and never changes afterwards. Do not copy mine without checking yours.

Under BIOS, only the boot block changes, and /boot-fallback disappears:

  boot.loader.grub = {
    enable = true;
    devices = [
      "/dev/disk/by-id/nvme-SAMSUNG_XXXX_YYYY"
      "/dev/disk/by-id/nvme-SAMSUNG_XXXX_ZZZZ"
    ];
  };

  boot.swraid = {
    enable = true;
    mdadmConf = ''
      ARRAY /dev/md0 metadata=1.2 name=boot
      MAILADDR root
    '';
  };

  fileSystems."/boot" = {
    device = "/dev/md0";
    fsType = "vfat";
    options = [ "fmask=0077" "dmask=0077" ];
  };

The MAILADDR line is not decorative. Without it, the mdadm module warns:

mdadm: Neither MAILADDR nor PROGRAM has been set. This will cause the `mdmon` service to crash.

With it, both configurations above evaluate without a single warning.

If you add GNOME, mind the renamed options:

  services.xserver.enable = true;
  services.displayManager.gdm.enable = true;
  services.desktopManager.gnome.enable = true;
  environment.systemPackages = with pkgs; [ gnome-tweaks gnome-terminal ];

The old paths under services.xserver. still work but warn, and the packages no longer live under gnome., the prefix having been flattened.

Installing

nixos-install

Then, before rebooting:

cat /proc/mdstat                      # BIOS only
ls -la /mnt/boot/EFI/BOOT/            # UEFI, on both ESPs
ls -la /mnt/boot-fallback/EFI/BOOT/

Those checks are useful but weak, and this is the point I would most like to land. grub-install writes into a mounted filesystem, and Linux mounts perfectly well what no firmware will ever read, since it has the md driver. Against the RAID ESP from the first section:

exit code: 0
Installation finished. No error reported.

The files are there, and they are real files. Nothing in the install path is firmware, so nothing has any reason to ask whether a firmware would reach those bytes. The installer verifies that the write happened. The question was who would be able to read it back.

The only check that counts is a boot with one disk unplugged. Do it on installation day, while there is nothing to lose.

What actually happens when a disk dies

Redundancy fails at two independent layers. The first, firmware and bootloader, is settled by everything above. The second is btrfs, and it is nastier.

A two-device btrfs RAID 1 refuses an ordinary mount when a disk is missing. The guide’s root options, subvol=@,compress=zstd,noatime, carry no degraded. The word appears three times across its pages, every time as a status to observe, never as the option that lets you boot.

And the rescue is less accessible than it used to be. The systemd initrd is on by default, and emergency access in it is closed by default:

emergencyAccess
  Set to true for unauthenticated emergency access, and false or
  null for no emergency access.
  default = false

Checked on the installed machine, booted with a single disk. The firmware started the bootloader, the boot layer held. Then:

Dependency failed for /sysroot.
Dependency failed for Initrd Root File System.
[  OK  ] Started Emergency Shell.
[  OK  ] Reached target Emergency Mode.
You are in emergency mode. After logging in, type "journalctl -xb" to view
Cannot open access to console, the root account is locked.

That last line is the whole problem. The emergency shell exists, and it is out of reach.

degraded in the options is not enough

This is where I had to correct what I was about to publish. Since options merge, adding degraded is a one-liner, and it is what everybody suggests:

  fileSystems."/".options = [ "degraded" ];

I applied it to the machine, rebuilt with nixos-rebuild boot, and rebooted with a single disk. It did not boot. Here is why:

         Expecting device /dev/disk/by-uuid…64d3-6496-4908-9a0d-6f0e3dc84948...
[ TIME ] Timed out waiting for device /dev/…a364d3-6496-4908-9a0d-6f0e3dc84948.
[DEPEND] Dependency failed for /sysroot.

The ellipses are systemd’s own, shortening over-long unit names. The mount is never attempted. udev publishes a btrfs filesystem’s by-uuid link only once all its members are present: systemd waits for a node that will never appear, and the option it would have used plays no part. It is correct and arrives too late.

I tried the obvious escape, naming the root by a partition rather than by the UUID, using the very lib.mkForce the error message had suggested. That fails too, for an independent reason: the initrd does not create by-id links.

         Expecting device /dev/disk/by-id/virtio-DISKA-part2...
[ TIME ] Timed out waiting for device /dev/disk/by-id/virtio-DISKA-part2.

And that disk was the one still attached.

No one-line option makes this machine reboot on its own on a single disk. The choice is not between "unattended" and "by hand", but between "by hand, prepared" and "by hand, locked out".

What does work, and what I measured

Only one line really counts, and it is the one that lets you in:

  boot.initrd.systemd.emergencyAccess = "$y$j9T$...";

A hashed password, or true for an unauthenticated shell if the machine is physically out of reach. The same single-disk boot then hands you a shell, and recovery takes two commands:

# btrfs device scan
Scanning for Btrfs filesystems
registered: /dev/vda2

# mount -o degraded,subvol=@ /dev/vda2 /m
RC=0

# ls /m
bin   boot  boot-fallback  dev  etc  home  lib  nix
proc  root  run  srv  sys   tmp  usr  var

# btrfs filesystem show /m
Label: 'nixos'  uuid: 05a364d3-6496-4908-9a0d-6f0e3dc84948
    Total devices 2 FS bytes used 1.24GiB
    devid    1 size 15.50GiB used 3.26GiB path /dev/vda2
    devid    2 size 0 used 0 path  MISSING

The volume mounts, the data is there, btrfs names the absent disk.

So: "the system boots even if one drive fails" is false on this layout. Booting survives, the data survives, getting back up needs a keyboard. That is already a great deal, and very different from what the guide announces.

A piece of folklore not to repeat, and I was about to repeat it: mounting a degraded btrfs RAID 1 read-write does not condemn the volume. The single block groups appear, but the second degraded mount still succeeds on kernel 7.1.3. Current kernels assess tolerance block group by block group.

Replacing the disk

Partition the new disk identically, redefine DATA_B for your mode, then treat the two layers separately.

Booting, under UEFI: format the new ESP with the right label, mount it, let NixOS populate it.

mkfs.fat -F 32 -n ESP-B /dev/disk/by-id/<new>-part1
mount -o fmask=0077,dmask=0077 /dev/disk/by-label/ESP-B /boot-fallback
nixos-rebuild boot

Under BIOS, mdadm does the rebuilding:

mdadm --manage /dev/md0 --add /dev/disk/by-id/<new>-part2
watch cat /proc/mdstat
nixos-rebuild boot

Then the data. The guide proposed btrfs device remove followed by device add. That first command cannot succeed on a two-disk array, in either state:

The command that does exist is called btrfs replace, and the guide never mentions it.

btrfs filesystem show /            # read the missing disk's devid
btrfs replace start <devid> "$DATA_B" /
btrfs replace status /

The devid, not the path: the documentation notes that this form becomes necessary as soon as the source device is disconnected, which is exactly the case for a dead disk. The target device must be the same size or larger.

Its closing balance is needed, just not where it put it: single block groups survive a replacement.

profiles immediately after the replace:
   Data, single: total=963.81MiB, used=250.78MiB
   Data, RAID1:  total=1.52GiB,   used=549.22MiB

after balance -dconvert=raid1 -mconvert=raid1:
   Data, RAID1:     total=980.00MiB, used=800.00MiB
   Metadata, RAID1: total=256.00MiB, used=1.16MiB
   System, RAID1:   total=32.00MiB,  used=16.00KiB

So, to finish:

btrfs balance start -dconvert=raid1 -mconvert=raid1 /
btrfs filesystem usage /

Right instinct, wrong sequence.

Monitoring

Three things are enough, already in the configuration above. The weekly autoScrub verifies checksums and repairs from the healthy copy. smartd watches disk health. Under BIOS, MAILADDR brings into existence the service that warns when the array degrades.

Under UEFI there is no equivalent, for want of an md array: a missing btrfs device announces itself only through btrfs device stats or the scrub. Monitoring therefore matters more here than elsewhere, since losing a disk stays invisible until the next boot.

None of these tools will tell you whether the machine would boot. Set yourself a reminder to unplug a disk once a year.

What I verified, and how

On a Mac with no Linux builder, across four tiers of increasing cost, plus reading the nixpkgs sources wherever the claim is about what NixOS does rather than what the hardware answers.

The fourth tier is a real install: the official NixOS 26.05 ISO for aarch64, a QEMU machine with two blank disks and EDK2 firmware, this guide worked through command by command up to nixos-install, then reboots with disks unplugged. That tier found the three most important corrections in this article, two of them about what I had written. The method has its own article.

The honest limit is the architecture. The real install and the grub-install measurements ran on aarch64-linux, the only native option on this machine, and the bootloader there is called BOOTAA64.EFI rather than BOOTX64.EFI. Nothing at stake here depends on that: mirroredBoots, the merging of mount options, systemd waiting on a by-uuid node and btrfs’s own behaviour are the same code on both sides. The superblock offset and BIOS mode were measured on x86.

Untested is the same exercise on physical hardware, with a motherboard firmware rather than EDK2, and a disk that really dies rather than a virtual cable pulled.

Sources

• mdadm(8), the --metadata section, for where the superblock goes and what --create defaults to
• Software RAID, kernel administration guide
• btrfs replace and Volume management
• NixOS’s GRUB module, for mirroredBoots and its assertions
• NixOS option search, to check names before writing them

The firmware behaviour described here is not cited: it is measured, under EDK2 and SeaBIOS.