[bitnami/etcd] Stop relying on files for state

[pckhoi]

Description of the change

The current etcd container and chart have a few major problems:

1. It relies on files outside of the data directory which could contain conflicting information compared to the data dir and the cluster
2. Removing the member during pre-stop hook is problematic. I guess this was added to support scaling down the cluster. If so, this logic is leaky. There are 2 cases where this breaks down:
   1. If the pod was killed for reasons other than replicas update then the next time the pod starts, it will not be able to start from the existing data dir which means it must throw away the data dir and start from scratch.
   2. If the cluster is scaled down and the PVC is retained, the next time the cluster is scaled up, the new member will encounter a non-empty data dir which it must discard
3. If the member was removed, the container chokes up on non-empty data dir in most cases except when recovering from a snapshot
4. It might attempt to add a new member even when an old member with the same name already exists. This is caused by relying on files for state
5. It runs etcdctl member update for unclear reasons when the data dir is not empty and there is a member ID
6. It relies on ETCD_INITIAL_CLUSTER_STATE to know whether the cluster is new which could be inaccurate

This PR add the following changes:

• Add preupgrade.sh which should be run in a Helm pre-upgrade hook. When the cluster is scaled down, it detects and removes obsolete members with etcdctl member remove.
• Remove prestop.sh
• Stop storing/checking member ID from the member_id file. Instead, the remote member ID is read from the cluster with etcdctl member list, and the local member ID is checked for conflict during startup.
• Stop storing/checking member removal state from member_removal.log. Check with etcdctl member list instead.
• If the data dir is not empty, check if the member still belongs to the cluster (remote ID and local ID are the same). If there is a conflict, remove the data dir, remove the old member, add a new member, and start the member from scratch.
• Remove environment variable ETCD_DISABLE_STORE_MEMBER_ID
• Remove environment variable ETCD_DISABLE_PRESTOP
• Environment variable ETCD_INITIAL_CLUSTER_STATE becomes read-only

Benefits

• Not relying on files outside of the data directory means there is only a single source of truths (or only as many as there are live members in the cluster plus the data dir), which makes most operations more reliable
• Removing obsolete members in Helm pre-upgrade hook means the etcdctl member remove command tends to be executed against a healthy cluster
• If the pod was killed for reasons other than replica changes, it can rejoin the cluster on its own while keeping all its data intact
• The container no longer chokes up on a non-empty data dir, even when the old member is removed

Possible drawbacks

• If during initialization there is a network outage and the current member can't connect to other members, it will think that it must start a new cluster. That said, I don't think there is any good solution in this case except manual recovery.
• I have not tested this set of changes outside of Helm/K8s

Applicable issues

• fixes #16069

Additional information

Related changes in the Helm chart: bitnami/charts#31161 and bitnami/charts#31164

[pckhoi]

I'm planning to open a complementary PR in the charts repo. I will try to add more tests there.

[juan131]

Hi @pckhoi

Thanks so much for this amazing contribution! It'd definitely help on making the Bitnami etcd chart more stable.

I think the main concern/challenge with your changes would be providing a solution for users who may scale down the cluster via kubectl scale sts/etcd --replicas X (or via some HorizontalPodAutoscaler that may also scale down the cluster without Helm's control via hooks). Correct me if I'm wrong but this use case won't be covered, right?

[pckhoi]

@juan131 you're correct that the autoscaling use case isn't covered. People use Etcd for its consistency rather than for handling large, fluctuating traffic so I think autoscaling to handle large traffic is a niche use case.

As for manual scaling, running helm upgrade makes more sense if the cluster is installed via Helm. If people are scaling with kubectl scale then they're probably not using Helm which makes it difficult to operate. So no, deploying/upgrading without Helm is also not supported.

[juan131]

Thanks for confirming so @pckhoi ! In that case, I'd add a warning at the "Upgrading" section alerting about what these changes imply (I mean, warning users to use exclusively Helm to scale the cluster):

• https://github.com/bitnami/charts/tree/main/bitnami/etcd#upgrading

We could even add it in the chart NOTES:

• https://github.com/bitnami/charts/blob/main/bitnami/etcd/templates/NOTES.txt

[pckhoi]

Sure, I will do that.

[pckhoi]

@juan131 I have updated https://github.com/bitnami/charts/tree/main/bitnami/etcd#upgrading. As for https://github.com/bitnami/charts/blob/main/bitnami/etcd/templates/NOTES.txt, I don't see anything that needs to be updated.

[pckhoi]

Thanks! I have addressed all the suggestsions.

[juan131]

@pckhoi I think this PR looks great now! Could you please check my comments in the associated chart PR? Thanks in advance.

[juan131]

While testing, I noticed that while increasing the number of replicas, the logs look like this:

INFO  ==> Current cluster members are: etcd-0
INFO  ==> Expected cluster members are: etcd-0,etcd-1,etcd-2

I'd include some message when we detect expected cluster member are greater or equal to the current ones. Sth like:

info "No obsolete members to remove. Pre-upgrade checks completed!"

Similar to this, while scaling down the logs look like this:

INFO  ==> Current cluster members are: etcd-0
INFO  ==> Expected cluster members are: etcd-0,etcd-1,etcd-2

I'd also add some extra messages so the logs are sth like:

INFO  ==> Current cluster members are: etcd-0,etcd-1,etcd-2
INFO  ==> Expected cluster members are: etcd-0,etcd-1
INFO  ==> Removing obsolete member etcd-2
Member 9fa135f920e34edd removed from cluster e111ddc420dc1d22
info "Pre-upgrade checks completed!"

[juan131]

Should we add some "retry" mechanism so we gave up on listing members after 3 consecutive failures or sth like that (check retry_while examples in the libetcd.sh library). This could help making the script more reliable.