Replies: 6 comments 16 replies
-
|
Usually if you don't see errors from scrub but do see them at runtime, it means they're from decrypting things, since scrub doesn't try decrypting things or else it wouldn't be able to scrub without unlocking things. There were, historically, a couple of cases of things that could cause decryption errors spuriously, but I think those should all be fixed by 2.1.11. There's counters in like, /proc/spl/kstat/kcf/ for how many times decryption failed: so you can see that that's what's going on. As far as getting errors with send/recv, if you try using -e from an unencrypted dataset to an encrypted receive, that'll fail, since embedded_data records can't be stored on native encryption datasets. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks! It looks like there were probably some decryption errors: If those are non-spurious decryption errors, what could be the cause of that? Some hardware/software bug introducing errors between encryption and checksum calculation? Hardware/software bug in the decryption code? |
Beta Was this translation helpful? Give feedback.
-
Hello, |
Beta Was this translation helpful? Give feedback.
-
|
The issue reveals itself when syncoid kicks in (and when the uptime is greater then approx 10days in my use case) Hereafter what I found in the log: |
Beta Was this translation helpful? Give feedback.
-
|
I think I can confirm that the errors are uptime-related. After 6 days uptime I only have one error reported. I'll see if the error frequency increases. |
Beta Was this translation helpful? Give feedback.
-
|
At 7 days 20 hours, I'm up to 7 errors listed. I'm going to write a script which tracks the appearance of errors over time. |
Beta Was this translation helpful? Give feedback.
-
|
I'm trying to find where in the code the |
Beta Was this translation helpful? Give feedback.
-
|
You can't find it because it was deleted for being "useless" in #12901, so I guess maybe it'll have a different name in 2.2, or just be gone entirely. Who can say. e: Having just checked, no, it's just deleted. Good thing nobody ever tries to debug encryption bugs, or else that might be a shame. |
Beta Was this translation helpful? Give feedback.
-
|
Here's the script I'm using to log when zpool errors appear: https://gist.github.com/mattico/d89172579cd69a4d8b8077c2e4fe8c17 I suppose it could be useful to also log |
Beta Was this translation helpful? Give feedback.
-
|
If it's a decryption error, it won't be from scrub - scrub doesn't try decrypting the blocks, ever, because then you couldn't scrub if you didn't have the key loaded. It might be caused by scrub running at the same time, somehow, but scrubbing itself should never try decrypting anything. |
Beta Was this translation helpful? Give feedback.
-
|
My suspicion , thinking about this a bit, would be if hypothetically, there were a case with decryption where reads timing out was incorrectly treated as a decryption error. I don't know at the moment of any such bug existing, but that would be what I would say the shape of this sounds like. |
Beta Was this translation helpful? Give feedback.
-
|
Since my last server restart, I changed two things:
My uptime is 18 days, which is the best uptime since I started to use encryption dataset. Might be just a coincidence though.... |
Beta Was this translation helpful? Give feedback.
-
|
@ofthesun9 Interesting datapoint. It's not easy for me to update my ZFS version because I'm using Debian Bullseye which doesn't have updated packages yet. I also use a syncoid service (on a 15 minute timer) which executes multiple syncoid processes one after the other. I'll try increasing the timer period and inserting some sleeps between the syncoid calls. It does seem that the errors are caused by (or noticed by) the syncoid process running, but I can't imagine why that would be. Maybe there's a kernel rate-limit on creating new KCF contexts? I can't find anything like that but I guess many things can act as unintentional rate limits. I'm not sure why zfs send would be any different than any other file read, but I guess it could end up reading all of the dataset data when most processes do not. |
Beta Was this translation helpful? Give feedback.
-
|
I would assume it's not a direct rate limit but something like an allocation failure being treated as a decryption failure or some similar outcome, I'm not really sure. zfs send isn't really like a file read, it's walking the metadata and finding all the metadata and data things that changed since the last incremental (or at all, for a non-incremental), and spitting them out. |
Beta Was this translation helpful? Give feedback.
-
|
Okay, this time I'm running kernel |
Beta Was this translation helpful? Give feedback.
-
|
Another odd symptom: my sanoid service has been stuck trying to take a snapshot for 1 day 11h:
|
Beta Was this translation helpful? Give feedback.
-
|
Huh, here's something new in the dmesg. Looks like a ZFS kernel task timed out and panic'd? In the console it additionally says That must be what's causing the hang. |
Beta Was this translation helpful? Give feedback.
-
|
Good thing native encryption gets bug fixes so reliably and fast, huh. |
Beta Was this translation helpful? Give feedback.
-
|
Another new development: errors are being detected at offsets other than 0x0 and 0x1. 0x20 is pretty popular, and there are some others: |
Beta Was this translation helpful? Give feedback.
-
|
For datasets, those aren't offsets, they're object IDs. |
Beta Was this translation helpful? Give feedback.
-
I have a pair of identical servers running Debian 11 with kernel
6.1.0-0.deb11.6-amd64and OpenZFS2.1.11-1~bpo11+1. They have an SSD mirror root pool (rpool) and a RAIDZ2 data pool (tank). I recently re-created both pools from snapshots to enable native encryption. Shortly after I noticed that one of the tank pools started reporting permanent errors:From what I've seen, all of the errors have been in snapshots or
<hex numbers>which I assume are deleted snapshots. (Deleting the snapshots turns them into hex numbers in the list) I've never seen any errors reported in any of the pool devices, nor in any files. I've never seen a ZFS scrub report any errors. I think the errors I have seen have been in just two specific datasets:tank/wordpressandtank/caddy.I was able to get all the errors to disappear by doing the following:
zpool clear tank(I'm not sure if this is necessary)zpool scrub tankand wait for it to complete.zpool scrub tankand wait for it to complete.(It's supposed to take two scrubs for reported errors to get cleared from what I understand)
I thought that had fixed the issue but after re-enabling syncoid, the errors started re-appearing. When syncoid attempts to send/recv some snapshots (in
tank/wordpressortank/caddy) it gets an I/O error.Update:
I can confirm that attempting to
zfs sendthe snapshots (with syncoid) is what causes the errors to be detected and appear inzpool status. At the moment thetank/caddydataset is fine buttank/wordpressandtank/nextcloud/dataare failing. I also get a new error:cannot receive incremental stream: kernel modules must be upgraded to receive this stream.which is weird because I confirmed that both kernel modules are the same version. Perhaps it's just data corruption causing that.Update:
I did a few cycles of running syncoid and deleting those snapshots which had errors and I got syncoid to complete successfully. So for the moment every dataset is able to create and send snapshots without error.
Update:
The errors started appearing again shortly after.
Are these errors real? Why doesn't scrub find them?
What more can I do to continue diagnosing this? How can I fix this?
Beta Was this translation helpful? Give feedback.
All reactions