Send/Receive old snapshots

[GuillaumeSeren]

Hello,
I have been working on a setup where there 3 zfs backups servers, that receive incremental snapshot,
that is working fine, but when we need to replace one of the backup server, the initial import from an other may be long (days/weeks),
so it can get some delay and then loose the common snapshot to work with incremental send (the client only save like 7d of snapshots).

TL;DR
So I would like to know if it is there a way to send/replace old snapshot that way I can start using the server without waiting and then sync missing snapshots afterward.

Details:
We start initialising a server by zfs send -R from old server, that way the new server is close enought to get a common snapshot.
If the peer is new and there is no existing snapshot so we start by zfs send -I to get everything on the backup server.
Then every hour or so we zfs send -i snapshot-previous-common snapshot-last-client that way it would be up-to-date.

Even if the backup would be fast enought, to not take that long,
there still the case of a failed send at some point and we want to be consistent,
and try to be sure that a given snapshot is the same on all servers (parent, name and content).

The issue is to send the missing previous snapshot, if there is a way to do it,
we could just run the backup start with zfs send -I , then roll with zfs send -i (that way it would start sooner),
also if we found some missing snapshot we send it to be consistent on all backup servers.

Best,
Guillaume.

[rincebrain]

You don't have to just zfs send -R foo/bar@snap30 and wait for it to send all of snap1 through snap30, you can just zfs send foo/bar@snap1 and then once that finishes, zfs send -i foo/bar@snap1 foo/bar@snap2 and so on.

You can also use bookmarks to keep a consistent "from" to send from, and/or holds to prevent the older snapshots from being purged accidentally while synchronizing.

I think your question needs a lot more detail before we can understand the problem you're encountering, tbh.

[GuillaumeSeren]

Thank you @rincebrain ,
yes I aggree this use-case need more details to be precise, I will add a more complete case,
but my question was to send older backup on an existing dataset maybe it is not possible but I am not sure.

I added a more complete description of my case.

[rincebrain]

You can't insert an old snapshot retroactively in a dataset, no. You could just send the latest one for now and then receive the older ones to a second dataset, then eventually switch which one you were using, but no, you can't, say, receive snap30 and then insert snap1 before snap30 later in place.

[GuillaumeSeren]

Thank you @rincebrain,
I was hoping for a secret option to do it.

But maybe there no way around the initial sync (zfs send -R)

[rincebrain]

As I said, you could just do zfs send -R foo/bar@snap1 and then -R -i snap1 snap3 and so on, and you could use bookmarks to make sure you always have a shared incremental "from" you can send from even if you delete the snapshot on the source, but no, generally not.

I often describe a whole lump of constraints ZFS hard assumes deeply as "ZFS doesn't change the past".

[GuillaumeSeren]

That answer my question thank you @rincebrain

[bghira]

do you mean, to remove old snapshots and still be able to use them as incremental source? you can create bookmarks before you remove the snapshots on the source and then this is possible to reduce the total burden of send -R. you can not send those removed snapshots, though. they must already exist on the destination.

[rincebrain]

You cannot currently use -R with bookmarks. It could be done, but wasn't initially, and nobody has since.

[GuillaumeSeren]

Not remove but add missing ancestors that way I could send last snapshot fast and complete old missing snapshot later