Pod is not mutated when reading optional secrets fails

[razvan-agape]

Environment configuration

• kube-secrets-init version: 0.5.0
• secrets-init script version: 0.5.0
• helm chart version: 0.9.3
• ClusterRole updated to deny get secrets - due to security requirements, we denied this permission
• MutatingWebhook failurePolicy: Fail
• log level: debug
• provider: google

Test manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: svcc
  name: svcc
  namespace: secretspoc
spec:
  replicas: 1
  selector:
    matchLabels:
      app: svcc
  strategy: {}
  template:
    metadata:
      labels:
        app: svcc
    spec:
      containers:
      - image: alpine:3.18.4
        name: second
        command:
        - /bin/sh
        - -c
        - |
          while true
          do
            date
            env | grep $FILTER
            echo "====="
            sleep $SLEEP_DURATION
          done
        resources:
          requests:
            cpu: "150m"
            memory: "150Mi"
          limits:
            cpu: "150m"
            memory: "150Mi"
        env:
        - name: EXIT_EARLY
          value: "true"
        - name: FILTER
          value: "GIT"
        - name: SLEEP_DURATION
          value: "10"
        - name: GIT_PASSWORD
          value: gcp:secretmanager:<password-secret>/versions/latest
        - name: GIT_SSH_KEY
          valueFrom:
            secretKeyRef:
              name: svcc-secrets
              key: GIT_SSH_KEY
              optional: true
        envFrom:
        - configMapRef: # GIT_USERNAME
            name: svcc-configmap
            optional: true
      - image: alpine:3.18.4
        imagePullPolicy: Always
        command:
        - /bin/sh
        - -c
        - |
          while true
          do
            date
            env | grep $FILTER
            echo "====="
            sleep $SLEEP_DURATION
          done
        name: first
        resources:
          requests:
            cpu: "150m"
            memory: "150Mi"
          limits:
            cpu: "150m"
            memory: "150Mi"
        env:
        - name: EXIT_EARLY
          value: "true"
        - name: FILTER
          value: "GIT"
        - name: SLEEP_DURATION
          value: "10"
        - name: GIT_PASSWORD
          value: gcp:secretmanager:<password-secret>/versions/latest
        - name: GIT_USERNAME
          valueFrom:
            configMapKeyRef:
             name: svcc-configmap
             key: GIT_USERNAME
        envFrom:
        - secretRef: # ssh-key
            name: svcc-secrets
            optional: true
      serviceAccountName: <sa-name> #sa had permission to read secrets from GCP SM
status: {}
---
apiVersion: v1
data:
  GIT_SSH_KEY: ZHVtbXkK # dummy secret, not a reference to GCP SM
kind: Secret
metadata:
  name: svcc-secrets
  namespace: secretspoc
---
apiVersion: v1
data:
  GIT_USERNAME: gcp:secretmanager:<name>/versions/latest
kind: ConfigMap
metadata:
  name: svcc-configmap
  namespace: secretspoc

Problem

Because kube-secret-init cannot read any kubernetes secrets (as configured in the RBAC) and, the pod mutation is blocked regardless of:

• the optional: true set in the pod manifest (for both envFrom and env:secretKeyRef)
• other env vars (from env:value and configmaps) referencing GCP SM secrets

Plus, the logs of the kube-secrets-init are not specific about any of this. The only log line is: level=debug msg="no pod init containers were mutated".
Only the ReplicaSet controller shows some information:

Warning  FailedCreate      66s                replicaset-controller  Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": failed to call webhook: an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"6ebe4fad-6f30-49d1-8bf0-b2880ff77b8a\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : failed to mutate containers for pod : failed to look for envFrom: failed to get secret secretspoc/svcc-secrets: failed to get secret secretspoc/svcc-secrets: secrets \\\"svcc-secrets\\\" is forbidden: User \\\"system:serviceaccount:secretspoc:kube-secrets-init\\\" cannot get resource \\\"secrets\\\" in API group \\\"\\\" in the namespace \\\"secretspoc\\\"\"}}}") has prevented the request from succeeding

Warning  FailedCreate      30s (x5 over 66s)  replicaset-controller  (combined from similar events): Error creating: Internal error occurred: failed calling webhook "pods.kube-secrets-init.admission.doit-intl.com": failed to call webhook: an error on the server ("{\"kind\":\"AdmissionReview\",\"apiVersion\":\"admission.k8s.io/v1beta1\",\"response\":{\"uid\":\"e5d08f29-bd31-40ca-9c58-d2c027f5472f\",\"allowed\":false,\"status\":{\"metadata\":{},\"status\":\"Failure\",\"message\":\"could not mutate object: failed to mutate pod: : failed to mutate containers for pod : failed to look for envFrom: failed to get secret secretspoc/svcc-secrets: failed to get secret secretspoc/svcc-secrets: secrets \\\"svcc-secrets\\\" is forbidden: User \\\"system:serviceaccount:secretspoc:kube-secrets-init\\\" cannot get resource \\\"secrets\\\" in API group \\\"\\\" in the namespace \\\"secretspoc\\\"\"}}}") has prevented the request from succeeding

Expectations

• It would be nice if the mutation would happen when references to GCP SM secrets are encountered, regardless of any errors related to kubernetes secret/configmap retrieval.
• It would be really useful to provide better logging about the behaviour/events-flow of the kube-secrets-init.