How to enable resource recreation

[oleksii-kalinin]

Hey, I'm playing around with os-operator and at some point caught an error:

{"level":"error","ts":"2024-01-17T15:02:14.175Z","msg":"Reconciler error","controller":"opensearchcluster","controllerGroup":"opensearch.opster.io","controllerKind":"OpenSearchCluster","OpenSearchCluster":{"name":"opensearch-cluster","namespace":"elastic-stack-logging"},"namespace":"elastic-stack-logging","name":"opensearch-cluster","reconcileID":"8d0f375d-c096-4181-8930-88a0ce29132c","error":"recreating object on immutable field change has to be enabled explicitly through the reconciler options: StatefulSet.apps \"opensearch-cluster-warm\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden","errorVerbose":"StatefulSet.apps \"opensearch-cluster-warm\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden\nrecreating object on immutable field change has to be enabled explicitly through the reconciler options\ngithub.com/cisco-open/operator-tools/pkg/reconciler.(*GenericResourceReconciler).ReconcileResource\n\t/go/pkg/mod/github.com/cisco-open/[email protected]/pkg/reconciler/resource.go:515\nopensearch.opster.io/pkg/reconcilers.(*ClusterReconciler).reconcileNodeStatefulSet\n\t/workspace/pkg/reconcilers/cluster.go:379\nopensearch.opster.io/pkg/reconcilers.(*ClusterReconciler).Reconcile\n\t/workspace/pkg/reconcilers/cluster.go:121\nopensearch.opster.io/controllers.(*OpenSearchClusterReconciler).reconcilePhaseRunning\n\t/workspace/controllers/opensearchController.go:321\nopensearch.opster.io/controllers.(*OpenSearchClusterReconciler).Reconcile\n\t/workspace/controllers/opensearchController.go:142\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226"}

So how I can enable resource recreation?

[itsmjeu]

SAme problem after enabling the perNode certificates, now I cant go back and I'm stuck with similar error

[swoehrl-mw]

Hi @oleksii-kalinin and @itsmjeu. In case the operator ever gets stuck in this situation you can perform the following steps to recover:

1. Temporarily stop the operator so it does not interfere: kubectl scale deployment opensearch-operator-controller-manager --replicas=0
2. Delete the opensearch statefulset without deleting the pods: kubectl delete statefulset <name-of-your-nodepool-statefulset> --cascade=orphan (repeat that for each nodepool in your opensearch cluster)
3. Restart the operator: kubectl scale deployment opensearch-operator-controller-manager --replicas=1
4. The operator will recreate the statefulsets and perform a rolling restart of the cluster pods to apply the new configuration. Depending on the configuration change the operator might not be able to perform the restart without downtime and you will need to delete all cluster pods to perform a full restart with downtime.