Keycloak container launch failed

[inlann]

Hello,

I'm trying to deploy theia-cloud in the kubernetes cluster created by kubeadm. I'm using theia-cloud 0.8.1.MS3 version and the keycloak chart version is 13.3.0. The terraform configuration for kubernetes cluster is here.

I didnot modify the keycloak configuration in the helm part. The keycloak pod cannot work:

tptuser@k8s-master:~$ kubectl -n keycloak get all
NAME                        READY   STATUS             RESTARTS      AGE
pod/keycloak-0              0/1     CrashLoopBackOff   1 (11s ago)   43s
pod/keycloak-postgresql-0   1/1     Running            0             43s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/keycloak                 ClusterIP   10.111.26.17   <none>        80/TCP     43s
service/keycloak-headless        ClusterIP   None           <none>        80/TCP     43s
service/keycloak-postgresql      ClusterIP   10.96.3.44     <none>        5432/TCP   43s
service/keycloak-postgresql-hl   ClusterIP   None           <none>        5432/TCP   43s

NAME                                   READY   AGE
statefulset.apps/keycloak              0/1     43s
statefulset.apps/keycloak-postgresql   1/1     43s
tptuser@k8s-master:~$ kubectl -n keycloak get all
NAME                        READY   STATUS             RESTARTS     AGE
pod/keycloak-0              0/1     CrashLoopBackOff   2 (2s ago)   56s
pod/keycloak-postgresql-0   1/1     Running            0            56s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/keycloak                 ClusterIP   10.111.26.17   <none>        80/TCP     56s
service/keycloak-headless        ClusterIP   None           <none>        80/TCP     56s
service/keycloak-postgresql      ClusterIP   10.96.3.44     <none>        5432/TCP   56s
service/keycloak-postgresql-hl   ClusterIP   None           <none>        5432/TCP   56s

NAME                                   READY   AGE
statefulset.apps/keycloak              0/1     56s
statefulset.apps/keycloak-postgresql   1/1     56s
tptuser@k8s-master:~$ kubectl -n keycloak logs keycloak-0 
keycloak 15:38:11.68 
keycloak 15:38:11.68 Welcome to the Bitnami keycloak container
keycloak 15:38:11.68 Subscribe to project updates by watching https://github.com/bitnami/containers
keycloak 15:38:11.68 Submit issues and feature requests at https://github.com/bitnami/containers/issues
keycloak 15:38:11.69 
keycloak 15:38:11.69 INFO  ==> ** Starting keycloak setup **
keycloak 15:38:11.70 INFO  ==> Validating settings in KEYCLOAK_* env vars...
keycloak 15:38:11.71 INFO  ==> Trying to connect to PostgreSQL server keycloak-postgresql...
keycloak 15:38:11.72 INFO  ==> Found PostgreSQL server listening at keycloak-postgresql:5432
keycloak 15:38:11.72 INFO  ==> Configuring database settings
/opt/bitnami/scripts/libfile.sh: line 39: /opt/bitnami/keycloak/conf/keycloak.conf: Permission denied

I have tried executing "terraform destroy" and "terraform apply" multiple times. Except for the pod "keycloak-0" encountering this permission error, sometimes the pod "keycloak-postgresql-0" also encounters a permission error:

tptuser@k8s-master:~$ kubectl -n keycloak delete pod keycloak-postgresql-0 
pod "keycloak-postgresql-0" deleted
tptuser@k8s-master:~$ kubectl -n keycloak get all
NAME                        READY   STATUS             RESTARTS        AGE
pod/keycloak-0              0/1     CrashLoopBackOff   7 (2m57s ago)   14m
pod/keycloak-postgresql-0   0/1     Init:0/1           0               5s

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/keycloak                 ClusterIP   10.111.26.17   <none>        80/TCP     14m
service/keycloak-headless        ClusterIP   None           <none>        80/TCP     14m
service/keycloak-postgresql      ClusterIP   10.96.3.44     <none>        5432/TCP   14m
service/keycloak-postgresql-hl   ClusterIP   None           <none>        5432/TCP   14m

NAME                                   READY   AGE
statefulset.apps/keycloak              0/1     14m
statefulset.apps/keycloak-postgresql   0/1     14m
tptuser@k8s-master:~$ kubectl -n keycloak get pod -w
NAME                    READY   STATUS             RESTARTS       AGE
keycloak-0              0/1     CrashLoopBackOff   7 (3m2s ago)   14m
keycloak-postgresql-0   0/1     Init:0/1           0              10s
keycloak-postgresql-0   0/1     Init:0/1           0              17s
keycloak-postgresql-0   0/1     PodInitializing    0              18s
keycloak-postgresql-0   0/1     Error              0              19s
keycloak-postgresql-0   0/1     Error              1 (1s ago)     20s
keycloak-postgresql-0   0/1     CrashLoopBackOff   1 (2s ago)     21s
^Ctptuser@k8s-master:~$ kubectl -n keycloak logs keycloak-postgresql-0 
Defaulted container "postgresql" out of: postgresql, init-chmod-data (init)
postgresql 15:51:23.86 
postgresql 15:51:23.86 Welcome to the Bitnami postgresql container
postgresql 15:51:23.86 Subscribe to project updates by watching https://github.com/bitnami/containers
postgresql 15:51:23.86 Submit issues and feature requests at https://github.com/bitnami/containers/issues
postgresql 15:51:23.86 
mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: Permission denied
mktemp: failed to create file via template ‘/tmp/tmp.XXXXXXXXXX’: Permission denied
/opt/bitnami/scripts/libpostgresql.sh: line 33: : No such file or directory

The pods keycloak-0 and keycloak-postgresql-0 do not encounter errors every time; sometimes they take turns having issues. For example, keycloak-postgresql-0 might fail to start due to permission errors, while the keycloak-0 pod is running and waiting for a connection to keycloak-postgresql. At other times, keycloak-postgresql-0 is running normally, while keycloak-0 encounters permission errors.

When the "keycloak-postgresql-0" encounters a permission error, I tried to start a container using that image with Docker. And that Docker container was able to start normally, and the permissions of the /tmp directory within it were correct.

I don't know how to fix this issue. It doesn't seem to be a problem with the image. I have previously deployed Keycloak normally using this method. However, to resolve a system disk space issue, I copied /var/lib/containerd to a /work directory mounted on another disk to address the problem of system disk space consumption. I created a symbolic link in /var/lib, where /var/lib/containerd is a symbolic link to /work/containerd. Keycloak began to report errors when redeploying theia-cloud after this change. However, when I tried to delete the symbolic link and move /work/containerd back to /var/lib to restore the environment, Keycloak's deployment still encountered this problem.

Any help? Really appreciate!

[jfaltermeier]

I haven't seen this before.

I can only imagine that maybe a security context in the postgress pod does not work well with the kubeadm created cluster or the Volume Provisioning for persistent volumes leads to wrong permissions.

The keycloak-postgresql-0 deployment uses a security context like this

      securityContext:
        runAsUser: 0

Also there are some volume mounts, but those don't look like they are affecting /tmp

      volumeMounts:
        - name: data
          mountPath: /bitnami/postgresql
        - name: dshm
          mountPath: /dev/shm
        - name: kube-api-access-llmmk
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
          
      volumeMounts:
        - name: dshm
          mountPath: /dev/shm
        - name: data
          mountPath: /bitnami/postgresql
        - name: kube-api-access-llmmk
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount

Since this is not an issue with Theia Cloud directly, maybe you can find better information/help here:
https://github.com/bitnami/charts/issues/
bitnami/charts#22990

If not you could probably try to create a hello world deployment testing some securityContext and volumeMounts to see if you can pin the issue down.