Replication Mover Pods Terminating with Rsync Error in TLS Mode - Configuration Issue

[Sanjeeth8733]

We are facing an issue with configuring the moverSecurityContext for both replicationSources and replicationDestinations in our Kubernetes environment. The goal is to enable Rsync-based replication in TLS mode for data transfer between the source and destination pods.

Current Configuration:
The required configuration for moverSecurityContext, which applies to the pods responsible for replication, is as follows:

`
podSecurityContext:
fsGroup: 1000
containerSecurityContext:
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true

`

Issue:
The replication mover pods are terminating with an Rsync error while attempting replication using TLS mode. The following log lines from the pod are relevant:

`
LOG7[2]: Local descriptor (FD=3) closed
LOG7[2]: Service [rsync] finished (1 left)
[73] rsync to data/ from UNDETERMINED (::ffff:10.131.0.2)
[73] receiving file list
[70] rsync: connection unexpectedly closed (17600 bytes received so far) [generator]
[70] rsync error: error in rsync protocol data stream (code 12) at io.c(228) [generator=3.2.3]
LOG7[3]: TLS alert (read): warning: close notify
LOG6[3]: TLS closed (SSL_read)

`

Attempted Solution:
We attempted to configure the moverSecurityContext with the provided settings, as shown below:

`
moverSecurityContext:
fsGroup: 1000
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true

`

However, the issue persists despite this configuration.

Additional Details:

The provided security context (fsGroup, runAsUser, runAsGroup, and runAsNonRoot) is applied to the source pod responsible for writing to the volume.
Source Operator Configuration:

`
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: src
namespace: doc
spec:
rsyncTLS:
moverSecurityContext:
fsGroup: 1000
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
address: 0.2.8.4
copyMethod: Direct
port: 00100
keySecret: tls-key-secret
sourcePVC: PVC-0
trigger: {}

`

The destination operator also has the same moverSecurityContext configuration:
Destination Operator Configuration:

`
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
name: destination
namespace: doc
spec:
rsyncTLS:
moverSecurityContext:
fsGroup: 1000
runAsUser: 1000
runAsGroup: 1000
runAsNonRoot: true
copyMethod: Direct
destinationPVC: PVC-0
serviceType: ClusterIP
keySecret: tls-key-secret

`

The replication is performed using the "Direct" method.
Please help us identify the root cause of the Rsync error in TLS mode and provide any possible solutions or suggestions to resolve this issue, I have also attache log file below. Thank you!

Source.log
Destination-1.log
Destination-2.log

[tesshuflower]

@Sanjeeth8733 I'm not sure anything particular is jumping out at me straight away with this issue.

Some questions I guess, to help narrow things down:

• Did you have rsync working in any configuration prior to this?
• I notice you're using clusterIP for the ReplicationDestination service. Is this an environment where the cluster IP is shared between multiple clusters? I'm assuming the ReplicationSource is in a separate cluster and then it's accessing the clusterIP from the destination cluster?

[Sanjeeth8733]

@tesshuflower Sure, applying the annotation volsync.backube/privileged-movers: "true" to the namespace worked, but the ideal solution we want is to use moverSecurityContext with the securityContext of the application pods.

[dheerajjoshim]

@tesshuflower I can answer the question while @Sanjeeth8733 is away

1. The rsync-tls replication was working fine when we annotated the namespace with volsync.backube/privileged-movers: "true"
2. Yes. The replication is between the cluster. We have an nginx ingress listening for the mover services

We thought rather than adding the annotation to the namespace we will set the moverSecurityContext with the application pods securityContext

[tesshuflower]

Interesting - was this switched from using the annotation to not?

If so it's possible some files were written as root when the replication was done with the namespace annotation and then a subsequent sync without privileged mode could have problems reading/writing the files.

Would it be possible to run a test replicationsource/dest with a new/different PVC on each side to see if that is successful?

[dheerajjoshim]

Yes. That's a possibility.

The ReplicationDestination and ReplicationSources were switched from using namespace annotation to moverSecurityContext without the application reinstallation or PVC cleanup.

We will check and report after using moverSecurityContext on a fresh deployment of the app

[Sanjeeth8733]

@tesshuflower

We wanted to share the results of the test replication you suggested. Here's what we observed during the process:

We followed your instructions carefully and set up a new/different PVC on each side for the test. The source was named "replicationsource," and the destination was "replicationdest."

To ensure smooth replication, we utilized the Mover Security Context, as advised.

Unfortunately, despite our efforts, the volumes did not sync successfully.

We made sure that the Pod UID and GID on both the source and destination matched those of the application Pod.

Here are the relevant error logs that were observed during the replication attempt:

destination-log.log
source-log.log

At this point, we are uncertain about the root cause of the syncing issue even with the Mover Security Context in place. If you have any further suggestions or ideas to explore, please let us know. I'm keen to resolve this matter and ensure successful replication going forward.

Thank you for your attention to this matter. Looking forward to your guidance.

[tesshuflower]

Unfortunately I don't have a good explanation at this point, a few thoughts though:

• is it possible that anything rsync is trying to copy from the disk is not owned by the UID/GID you're using?
• can you compare the pod security context seccompProfile value from your app pod and the replicationsource pod?
• We could potentially look at whether using direct is causing an issue (while the app pod is also using the disk at the same time). Do you have more info about the PVC access mode you're using? One thing you could run as a test is use a replicationsource/replicationdestination with copyMethod: Snapshot so that a separate PVCs are created to read/write to from your app, just to rule out an issue there.

[dheerajjoshim]

• rsync isn't trying to copy any directories outside the ones mounted by the PVCs
• Pod seccompProfile is RuntimeDefault. When it is RuntimeDefault should the same be set for mover pods?
• App pod is using the disk only in the source, at the destination the pods are not running.

[tesshuflower]

• rsync isn't trying to copy any directories outside the ones mounted by the PVCs
• Pod seccompProfile is RuntimeDefault. When it is RuntimeDefault should the same be set for mover pods?
• App pod is using the disk only in the source, at the destination the pods are not running.

• Normally I would think the seccompProfile would get filled in automatically with RuntimeDefault - is this not happening on the mover pod? If so you may want to try setting it in the moverSecurityContext
• It might be worth trying a test without the app running (or use the snapshot method I mentioned above) to help narrow it down. In case there's some issue with app using the disk at the same time as the mover pod.

[dheerajjoshim]

We can try it