2FA Disk encryption with NixOS and ZFS, +1FA seperate encrypted user homes

This builds on the previously posted NixOS disk encryption with YubiKey 1FA/2FA: From scratch, USB bootable, potentially including swap.

Motivation

As discussed in the previous post, trusted computing on untrusted hardware (specifically keyloggers or side-channels that give info about key presses) is the expected threat model.

Locking a computer with a typical lock screen only locks the user account on the booted system. It does not secure the user data. On an encrypted disk, the disk encryption key is stored in RAM. Clearing the stored encryption key from RAM for an encrypted root partition is not possible, as a typical lock screen still requires the kernel and some other system components to continue operating. This means a potential attacker could read the disk encryption key from the booted system in lock screen, e.g. by freezing the RAM contents.

Further, solid root disk encryption does not protect against a rogue administrator. User-specific encryption would be needed for that.

Solutions to these problems exist. The simplest to understand is probably to seperate between a (potentially encrypted) root partition, and (multiple, potentially encrypted) user home partitions. The user home partitions would be linked to the users and only unlocked on login. And, importantly, could be unmounted and the key removed from RAM on logout/lock. Problem with traditional LUKS+LVM solutions on ext4: storage requirements for user home and have to be known ahead of time. It is not (trivially) possible to format the disk in a way where users dynamically share storage space with the rest of the operating system while their user home is unlocked with a different key.

Goals/Requirements

To achieve all these goals, we will be using ZFS inside a LUKS container unlocked with 2FA, and put user homes into zfs datasets in one big shared zpool. Datasets will dynamically share all the storage space in their pool, while being encryptable on a per-dataset level. Putting the zpool into a LUKS encrypted partition is necessary because systemd-boot does not directly support 2FA unlocking of ZFS native encryption.

Resources

None of these guides worked well, yielding half-broken installs or installs that’d combust after the first boot. Or just installs that did have encryption, but no yubikey support.

Further, some specific ZFS resources are helpful:

Getting started/Requirements:

This is identical to the other article. Nothing new yet.

nix-shell https://github.com/sgillespie/nixos-yubikey-luks/archive/master.tar.gz

Create the luks disk key

This is identical to the other article. Nothing new yet. This should be ran in the nix shell specifically prepared to deal with yubikey stuff.

SLOT=2 # either 1 or 2 works, depends on which slots of your yubikey are used


# optional; overrides the existing cryptographic key on that yubikey slot.
#  Making multiple installs with one yubikey, you might not want to override your slot
ykpersonalize -"$SLOT" -ochal-resp -ochal-hmac

SALT_LENGTH=16
SALT="$(dd if=/dev/random bs=1 count=$SALT_LENGTH 2>/dev/null | rbtohex)"

read -s USER_PASSPHRASE # Skip if you want only Yubikey 1FA
echo $USER_PASSPHRASE # check your password! Otherwise, you can pretty much restart from here.
# make sure to not compromise security by storing password in your shell history.

CHALLENGE="$(echo -n $SALT | openssl dgst -binary -sha512 | rbtohex)"

# this is what will happen on unlock to calculate the disk key
RESPONSE="$(ykchalresp -"$SLOT" -x $CHALLENGE 2>/dev/null)" # replace -2 with -1 if you used slot 1 on the yubikey

KEY_LENGTH=512 # if AES-256, use 512 key length. Other algos will have different.
ITERATIONS=1000000 # Make it hard to reverse by doing a bunch of iterations. Too much will make the unlock slow, too little will reduce cracking effort needed.


# run ONLY the one appropriate one!!
# 2FA (with user password):
LUKS_KEY="$(echo -n $USER_PASSPHRASE | pbkdf2-sha512 $(($KEY_LENGTH / 8)) $ITERATIONS $RESPONSE | rbtohex)"

# 1 FA (no user password):
LUKS_KEY="$(echo | pbkdf2-sha512 $(($KEY_LENGTH / 8)) $ITERATIONS $RESPONSE | rbtohex)"


CIPHER=aes-xts-plain64
HASH=sha512

Partitioning

Start by creating a formatted disk:

gdisk /dev/sda # can be ran in any environment that has gdisk

After doing these steps, the planned setup should look as follows:

Command (? for help): p
Disk /dev/sda: 1953525168 sectors, 931.5 GiB
Model: RTL9210 NVME
Sector size (logical/physical): 512/2048 bytes
Disk identifier (GUID): D894E466-C549-40DA-94E8-5503E3814FF7
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 1953525134
Partitions will be aligned on 2048-sector boundaries
Total free space is 3437 sectors (1.7 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1026047   500.0 MiB   EF00  EFI system partition
   2         1026048         3123199   1024.0 MiB  8300  Linux filesystem
   3         3123200      1953523711   930.0 GiB   8309  Linux LUKS

Command (? for help):

Don’t worry, we didn’t yet actually set up any root partition, that follows in the one LUKS container!

Formatting and setting up ZFS [the new stuff!]

After creating the partition, you’ll have to format them.

# using above defined shell variables to create the luks container:
echo -n "$LUKS_KEY" | hextorb | cryptsetup luksFormat --cipher="$CIPHER" \
  --key-size="$KEY_LENGTH" --hash="$HASH" --key-file=- /dev/sda3

# open the luks container:
echo -n "$LUKS_KEY" | hextorb | cryptsetup open /dev/sda3 encrypted --key-file=-

# create the zpool
zpool create
  -O atime=off \ # not storing read times of files (optional)
  -O compression=lz4 \ # i like compression (optional)
  -O mountpoint=none \ # zpool itself doesn't mount to anywhere
  -O xattr=sa \ # apparently good for data sets with small files
  -O acltype=posixacl \ # required for linux journal to work correctly
  -o ashift=14 \ # check for your SSD.
  zpool /dev/mapper/encrypted # initializing the dataset in the LUKS partition

# initialize datasets
zfs create -o mountpoint=legacy zpool/root
zfs create -o mountpoint=legacy zpool/nix
zfs create -o mountpoint=legacy zpool/var
zfs create -o mountpoint=legacy \
  -o canmount=off \ # global home doesn't actually mount
  -o org.openzfs.systemd:ignore=on \
  zpool/home

# this will prompt for a dataset password.
# Enter the same password as you plan to use for your user later.
zfs create -o mountpoint=/home/grimmauld \ # replace with your username
  -o encryption=on \ # home is encrypted
  -o keyformat=passphrase \ # key format is a passphrase.
  -o keylocation=prompt \ # only relevant when manually mounting.
  -o canmount=noauto \ # automatic mount fails on boot
  zpool/home/grimmauld # replace with your username

# format the non-zfs partitions
mkfs.fat /dev/sda1 -n BOOT
mkfs.ext4 -L salt /dev/sda2

# store salts, mount partitions
mkdir /mnt

mount -t zfs zpool/root /mnt
mkdir /mnt/nix /mnt/var /mnt/home /mnt/boot /mnt/crypt-storage
mount -t zfs zpool/nix /mnt/nix
mount -t zfs zpool/var /mnt/var
mkdir -p /mnt/home/grimmauld # replace with your username


mount /dev/sda1 /mnt/boot
mount /dev/sda2 /mnt/crypt-storage
echo -ne "$SALT\n$ITERATIONS" | tee /mnt/crypt-storage/default # store salt and iterations. tee because thats easier to sudo :3

Actually installing NixOS

nixos-generate-config --root /mnt

After generating the config, do the standard nix stuff like changing locale, maybe adding wifi, creating a user account etc etc.

When adding a user account, make sure you set the password of that user account to be the same as that users zfs partition encryption password.

Make sure your configuration uses systemd-boot:

boot.loader.systemd-boot.enable = true;
boot.loader.efi.canTouchEfiVariables = true;

Add kernel modules to allow inputs on systemd-boot screen to unlock the disk:

boot.kernelModules = [ "kvm-intel" "vfat" "nls_cp437" "nls_iso8859-1" "usbhid" "usb_storage" "nvme" ];

I added usb_storage and nvme modules because i run an external SSD via USB, nvme is for (modern) SSDs and usb storage is for USB. Depending on your hardware, you’d want to change it to sd_mod and/or ahci or similar, see https://gist.github.com/CMCDragonkai/810f78ee29c8fce916d072875f7e1751

Enable ZFS-related options:

boot.supportedFilesystems.zfs = true;
boot.zfs = {
  # user homes shouldn't be attempted to decrypt at boot. This'd store keys in RAM.
  forceImportRoot = false;
  requestEncryptionCredentials = false;
};

# this will unlock user homes on login, and lock on logout
security.pam = {
  zfs = {
    enable = true;
    homes = "zpool/home";
  };
};

boot.kernelPackages = config.boot.zfs.package.latestCompatibleLinuxPackages;
networking.hostId = "40fa5ea8"; # `head -c4 /dev/urandom | od -A none -t x4`

Add the mount points for your system:

fileSystems."/" =
  { device = "zpool/root";
    fsType = "zfs";
  };

fileSystems."/nix" =
  { device = "zpool/nix";
    fsType = "zfs";
  };

fileSystems."/var" =
  { device = "zpool/var";
    fsType = "zfs";
  };

Make systemd-boot use the yubikey:

  boot.initrd.luks.yubikeySupport = true; # enable yubikey support

  boot.initrd.luks.devices."root" = {
      device = "/dev/disk/by-uuid/e2e2c887-b7f8-4e47-b536-fe1722619836"; # /dev/sda3
      preLVM = true;
      allowDiscards = true;

      yubikey = {
        slot = 2;
        twoFactor = true; # Set to false for 1FA
        gracePeriod = 30; # Time in seconds to wait for Yubikey to be inserted
        keyLength = 64; # Set to $KEY_LENGTH/8
        saltLength = 16; # Set to $SALT_LENGTH

        storage = {
          device = "/dev/disk/by-uuid/541a42a8-3423-4e01-8006-6d50e67ed2b8"; # same ID as the crypt-storage mount earlier
          fsType = "ext4";
          path = "/default";
        };
      };
    };

Once everything is ready, run the install:

nixos-install

And remember to export the zpool, or you’ll have troubles booting:

zpool export zpool

Enjoy!

Making sure it actually works

Well, booting is one thing. You should log in, create a large file in your user home using dd if=/dev/urandom of=... bs=... count=... (or else compression might make it appear smaller), and invoke zfs list. Make sure the size of your user specific home dataset is the one that increased from this operation. If that is not the case, your user home does not live in the encrypted user home dataset and as such is visible from just unlocking the root disk encryption. So make sure it works!!! Setting this up was fiddely, testing is definitely required.