Un-encrypted Snapshots

[dbergloev]

Hi.

I recently started using ZFS's native encryption option. I have a large raidz setup where each disk is encrypted using dm-crypt and zpools are added to the un-encrypted device files. But I also have a volumes pool on a different disk that is used for all of the data that is used by the docker containers. This is not encrypted but I wanted a few of the datasets to be for things like the passbolt database container for an example and for this purpose I thought that ZFS's native encryption would be great. That way I did not have to slow things down to much by having the entire disk encrypted. Only a few datasets.

But I have found some issues with this. Most importantly the snapshot feature. What is the purpose of having an encrypted dataset, if all the snapshots that you create just lays there un-encrypted? I just recently found out when I had to restore all of the datasets from backup files after a server reinstall. Until then I thought it obvious that my snapshots was just as protected as the datasets they derived from.

Am I missing something?

[gmelikov]

Just use --raw arg with zfs send https://openzfs.github.io/openzfs-docs/man/8/zfs-send.8.html?highlight=raw#w

[dbergloev]

My problem was how the snapshots was stored, but I have done some more testing and it seams that I got tricked by ZFS's automatic nature.

The backup file

So I was sending snapshots to backup files on another storage device. Recently I required those files because of a complete reset of the original storage device. But after restoring it I found that all datasets, even the ones that had been encrypted, was now decrypted. This concerned me because I had no keyfile loaded at this time. I then found that the backup files was stored in decrypted state which by itself is not an issue seen as that entire storage device is full-disk-encrypted. My concern was more about how the snapshots was being stored after being created. Would ZFS store these as decrypted when taking snapshots of loaded datasets.

I then checked the current snapshots on the newly created ZFS device, and if you run zfs get keylocation on a snapshot from an encrypted dataset, you'll get VALUE=-. At this point it would seam that ZFS did not store this snapshot in an encrypted state.

More testing

I have spent a day looking for more information on this, but I could not find anything about how these snapshot are being stored or whether there is difference between creating snapshots from datasets with loaded or un-loaded keys. And there does not seam to be any actual ZFS output that would actually tell me the state of a snapshot. The only way to check encryption state would seam to be zfs get keylocation which only reported concerning info about the snapshots.

So I now tried a different test than my previous ones.

$ zfs create volumes/test -o encryption=on -o keyformat=passphrase
Enter new passphrase:
Re-enter new passphrase:

$ zfs get mounted volumes/test
NAME          PROPERTY  VALUE    SOURCE
volumes/test  mounted   yes      -

$ echo "This has been decrypted" | sudo tee /volumes/test/decrypted.txt
$ zfs snapshot volumes/test@snap1
$ zfs umount volumes/test
$ zfs unload-key volumes/test
$ zfs get keylocation volumes/test@snap1
NAME                PROPERTY     VALUE        SOURCE
volumes/test@snap1  keylocation  -            -

$ zfs send volumes/test@snap1 | sudo zfs recv volumes/test2
warning: cannot send 'volumes/test@snap1': dataset key must be loaded
cannot receive: failed to read from stream

I was relived to see the last two lines of this.

[slavanap]

@dk-zero-cool FYI -- here you pipe UNENCRYPTED data zfs send volumes/test@snap1
Use zfs send -w volumes/test@snap1 to pipe encrypted data (it works without a key loaded)

[dbergloev]

Thanks, good to know. But the RAID that i store them on is already encrypted with LUKS, so not important. I just need to remember to re-enable dataset encryption if I ever need to restore them at some point.

[dbergloev]

In conclusion snapshot seams to be stored as they should and the key is inherited from the actual parent dataset, but not reported as such. Because of this there does not seam to be any difference between creating snapshots from loaded or un-loaded datasets, which is also the expected behavior.

Sending the snapshot with zfs send will send it as decrypted by default, which is fine as long as you take this into account. But this is what caused my confusion to begin with as I had not even considered the snapshots to be a problem before this.