↓Skip to main content
  1. Main/
  2. How-To/

Proxmox with encrypted ZFS via Tang & Clevis

Table of Contents
Warranty There is no warranty that this setup works for any system which provide the dependencies listed below. I'm just providing these information because it worked for me. If you have questions you can leave a message here but I decide whether I'll answer and help or not.

Running a Proxmox VE server with encrypted ZFS for data requires decrypting and mounting the ZFS datasets manually after boot.

This can be improved by using a file with random binary data as keyfile. While this approach solves entering password it breaks the intention of encryption as the keyfile will be stored on an unencrypted system. Thus everyone with access to the system can use the keyfile and decrypt the datasets.

This is solved by an approach that uses cryptsetup to store the keyfile on an encrypted volume which is decrypted upon boot by entering a password. As Grub and systemd-boot support cryptsetup the decryption can be initiated directly during boot process and request entering the decryption password. That way the ZFS datasets and their keyfile are stored encrypted while still decrypting everything automatically.

The only problem left with this approach is the manual step required for booting the system by entering the password via IPMI or boot-time-SSH. As I wanted the system to boot without any manual step I developed the setup described on this page. It uses Clevis together with Tang to decrypt the encrypte keyfiles used by ZFS automatically upon boot.

Tang is a server for binding data to network presence. Clevis is a pluggable framework for automated decryption.

The Tang-Clevis decryption process is based on McCallum-Relyea key exchange.

Implementation #

I’m running Tang server on my Turris Omnia router. The following steps describe the client side implementation on my Proxmox VE server.

  1. install clevis

    apt install clevis

  2. export environment variables for simplicity (:!: change to meet your needs :!:)

    export keystore="/etc/keystore" # path to store encrypted ZFS keyfiles
export tmpdir="/tmp/zfs_clevis" # path to temporary store unencrypted ZFS keyfiles
export keyfile="home-user.key" # name of ZFS keyfile

  3. prepare helper vairables

    export keypath="${keystore}/${keyfile}"
export tmp_keypath="${tmpdir}/$(systemd-escape "${keypath}")"
export tmpdir_esc="$(systemd-escape --path "${tmpdir}")"

  4. create keyfile

    mkdir "${keystore}" "${tmpdir}"
curl "http://tang:8888/adv" > "${keystore}/tang-adv.jws"
mount -t tmpfs -o noatime,nosuid,nodev,noexec,mode=0700 none "${tmpdir}"
dd if=/dev/urandom of="${tmp_keypath}" bs=32 count=1
clevis encrypt tang '{"url":"http://tang:8888", "adv": "'"${keystore}"'/tang-adv.jws"}' < "${tmp_keypath}" > "${keypath}"

  5. create encrypted dataset

    zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation="file://${tmp_keypath}" rpool/home/user

  6. create systemd-service to mount ZFS

    cat > /etc/systemd/system/zfs-mount-network.service <<EOF
[Unit]
Description=Mount ZFS filesystems after network
Before=pve-guests.service
After=network.target
ConditionPathIsDirectory=/sys/module/zfs
#OnFailure=notify-email@%i.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zfs mount -a

[Install]
WantedBy=zfs.target
RequiredBy=pve-guests.service
EOF

    (OnFailure can call a service to notify you about errors)

  7. create systemd-service to load ZFS keys

    cat > /etc/systemd/system/zfs-load-key.service <<EOF
[Unit]
Description=Load encryption keys
DefaultDependencies=false
Before=zfs-mount-network.service
After=zfs-import.target

[Service]
Type=oneshot
ExecStart=/usr/bin/zfs load-key -a

[Install]
RequiredBy=zfs-mount-network.service
EOF

  8. create tmpfs mount unit for decrypted keys

    cat > "/etc/systemd/system/${tmpdir_esc}.mount" <<EOF
[Unit]
PartOf=zfs-load-key.service
StopWhenUnneeded=true

[Mount]
What=tmpfs
Where=${tmpdir}
Type=tmpfs
Options=noatime,nosuid,nodev,noexec,mode=0700
DirectoryMode=0700

[Install]
RequiredBy=zfs-load-key.service
EOF

  9. create systemd-service for keyfile decryption

    cat > /etc/systemd/system/clevis-decrypt@.service <<EOF
[Unit]
Description=Decrypt keyfiles using clevis
Before=zfs-load-key.service
After=network.target ${tmpdir_esc}.mount
Requires=${tmpdir_esc}.mount
ConditionPathExists=%I
ConditionFileNotEmpty=%I

[Service]
Type=oneshot
RemainAfterExit=no
ExecStart=/bin/bash -c "/usr/bin/clevis decrypt tang '{\"url\": \"http://tang:8888\", \"adv\": \"${keystore}/tang-adv.jws\"}' < %I > ${tmpdir}/%J"

[Install]
RequiredBy=zfs-load-key.service
EOF

  10. create systemd-unit to start pve-guests manually

    cat > /etc/systemd/system/pve-guests-restart.service <<EOF
[Unit]
Description=Force start of pve-guests
Requires=pve-guests.service

[Service]
Type=oneshot
ExecStart=/usr/bin/sleep 1
EOF

  11. activate systemd units

    systemctl daemon-reload
systemctl enable zfs-load-key.service "clevis-decrypt@${tmp_keypath}.service" zfs-mount-network.service

Backup #

To always be able to decrypt your ZFS volumes you need to backup the following information:

  • content of "${keystore}/${keyfile}" which is encrypted by clevis (example path /etc/keystore/home-user.key
  • content of all files in /var/db/tang/ on tang server

All these information must be backuped synchroniously, meaning that those files need to be from the same time as you cannot decrypt your ZFS otherwise and will loose the data.

↑