Help needed to write a mutation policy to modify the container.command[] and container.args[] by enforcing the call of a script.sh prior to any command and args.

[kikokikok]

Hi All,

I have been struggling to find the appropriate way to introduce a mutation policy which would allow me, given an annotation, to call a script.sh before any command and args provided in the manifests. It cannot be run as a init container as the need is to export ENV variables which are sensitive the the $HOME of the User running the container.

So far my Policy looks like this

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-aws-sso-env-vars
  annotations:
    policies.kyverno.io/title: Add AWS Environment Variables if Annotated
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/subject: Pod
    policies.kyverno.io.category: Other
    policies.kyverno.io.description: >-
      This policy generates a ConfigMap with an init script and mutates all initContainers (if present) and containers in a Pod with environment variables, but only if the Pod has the annotation `inject-aws-env: "true"`.
spec:
  rules:
    - name: generate-aws-env-init-script
      match:
        any:
          - resources:
              kinds:
                - Pod
              annotations:
                inject-aws-env: "true"
      generate:
        synchronize: true
        apiVersion: v1
        kind: ConfigMap
        name: aws-env-init-script
        namespace: "{{ request.object.metadata.namespace }}"
        clone:
          namespace: kyverno
          name: aws-env-init-script

    - name: add-init-container
      match:
        any:
          - resources:
              kinds:
                - Pod
              annotations:
                inject-aws-env: "true"
      mutate:
        patchStrategicMerge:
          spec:
            volumes:
              - name: aws-env-init-script
                configMap:
                  name: aws-env-init-script
              - name: aws-env
                emptyDir: {}
            initContainers:
              - name: set-aws-env
                image: amazon/aws-cli
                securityContext:
                  runAsUser: "{{ to_number(request.object.metadata.annotations.\"inject-aws-env-uid\") }}"
                  runAsGroup: "{{ to_number(request.object.metadata.annotations.\"inject-aws-env-gid\") }}"
                command: ["/bin/sh", "-c"]
                args:
                  - |
                    set -e
                    echo "Starting init container script"
                    ls -l /config
                    cp /config/init.sh /aws-env/init.sh
                    chmod +x /aws-env/init.sh
                    echo "Running init.sh script"
                    /aws-env/init.sh
                    echo "Init script completed"
                volumeMounts:
                  - name: aws-env-init-script
                    mountPath: /config
                  - name: aws-env
                    mountPath: /aws-env

    - name: inject-aws-env-vars
      match:
        any:
          - resources:
              kinds:
                - Pod
              annotations:
                inject-aws-env: "true"
      mutate:
        foreach:
          **- list: "request.object.spec.containers"
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    volumeMounts:
                      - name: aws-env
                        mountPath: /aws-env
                    command: ["/bin/sh", "-c", "/aws-env/aws-env.sh"]
                    args: ["{{ element.command[] }}", "{{ element.args[] }}"]**

I am struggling to get the args part right...
I am a total beginner on Kyverno and JMESPath and can't seem to find documentation or examples to help achieve it.
I have highlighted the problematic section in Bold as the rest of the policy is behaving as expected.

Thanks for your help !

[chipzoller]

Just to confirm, you're trying to take the commands from the incoming version of the container and apply them to args while at the same time overwriting the command[] list?

[chipzoller]

I recommend you confine this mutation to a CREATE request only by specifying operations[] under your match statement.

[kikokikok]

Once implemented, the policy is not deployable anymore:

admission webhook "validate-policy.kyverno.svc" denied the request: policy contains invalid variables: variable substitution failed for rule inject-aws-env-vars: variable [`null`,`null`][] must match regex "request|serviceAccountName|serviceAccountNamespace|element|elementIndex|@|images|image|([a-z_0-9]+\()[^{}]" or patterns [originalCmd* originalArgs*]

Thanks a lot for your time and sorry to bother you with my issues but I can't find any documentation or samples which would help me somehow figure it out

[chipzoller]

Ok, well maybe you should use a precondition in this rule so it only fires if it sees at least one container with a volumeMount for aws-env.

[kikokikok]

Well I have tried every potential option I could and I still am unable to understand what is wrong with the policy.
I have activated the debug mode and raised the log level to 6 and still I am unable to get the input supposed to be mutated so that I could at least understand what is going on.

Any suggestions on how to debug/troubleshoot failing mutate policies ?

[chipzoller]

What version of Kyverno are you on? And what are the contents of your present rule?

[kikokikok]

Chart Version: 3.2.6
App Version: 1.12.5

Policy is:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: call-aws-env-script
  annotations:
    policies.kyverno.io/title: Add AWS Environment Variables if Annotated
    policies.kyverno.io/minversion: 1.6.0
    policies.kyverno.io/subject: Pod
    policies.kyverno.io.category: Other
    policies.kyverno.io.description: >-
      This policy injects AWS environment variables into Pods annotated
      with `inject-aws-env: "true"`. It uses an init container to set up
      the environment and executes the original container command.
spec:
  background: false
  rules:
    - name: inject-aws-env-volume
      match:
        any:
          - resources:
              kinds:
                - Pod
              annotations:
                inject-aws-env: "true"
      mutate:
        foreach:
          - list: "request.object.spec.containers"
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    volumeMounts:
                      - name: aws-env
                        mountPath: /aws-env

    - name: inject-aws-env-vars
      match:
        any:
          - resources:
              kinds:
                - Pod
              annotations:
                inject-aws-env: "true"
      mutate:
        foreach:
          - list: "request.object.spec.containers"
            preconditions:
              any:
                - key: "{{ element.command[] | length(@) }}"
                  operator: GreaterThanOrEquals
                  value: 1
                - key: "{{ element.volumeMounts[?contains(@, 'aws-env')] }}"
                  operator: Equals
                  value: 'true'
            context:
              - name: originalCmd
                variable:
                  value:
                    data: "{{ element.command[] | `[]` }}" # Note: No need for [] here
              - name: originalArgs
                variable:
                  value:
                    data: "{{ element.args[] | `[]` }}"   # Note: No need for [] here
            patchStrategicMerge:
              spec:
                containers:
                  - name: "{{ element.name }}"
                    volumeMounts:
                      - name: aws-env
                        mountPath: /aws-env
                    command:
                      - /bin/sh
                      - -c
                      - /aws-env/aws-env.sh -- {{ originalCmd.data.join(' ', @) }} {{ originalArgs.data.join(' ', @) }}

Error seems related to the command[] being and failing to be interpreted by JMESPath expression

{
  "gvk": "/v1, Kind=Pod",
  "gvr": { "group": "", "version": "v1", "resource": "pods" },
  "namespace": "argocd",
  "name": "",
  "operation": "CREATE",
  "uid": "08d08d37-cc8d-41ac-b828-2454aed6e611",
  "user": {
    "username": "system:serviceaccount:kube-system:replicaset-controller",
    "uid": "bf226687-7c79-4823-8f5f-2e83af0806b2",
    "groups": [
      "system:serviceaccounts",
      "system:serviceaccounts:kube-system",
      "system:authenticated"
    ],
    "extra": {
      "authentication.kubernetes.io/credential-id": [
        "JTI=26ae5dbe-2698-4589-b3a7-ea132fd096e7"
      ]
    }
  },
  "roles": [],
  "clusterroles": [
    "system:basic-user",
    "system:controller:replicaset-controller",
    "system:discovery",
    "system:public-info-viewer",
    "system:service-account-issuer-discovery"
  ],
  "resource.gvk": "/v1, Kind=Pod",
  "kind": "Pod",
  "URLParams": "",
  "error": "mutation policy call-aws-env-script error: failed to apply policy call-aws-env-script rules [inject-aws-env-vars: failed to mutate elements: failed to evaluate mutate.foreach[1].preconditions: failed to substitute variables in condition key: failed to resolve element.command[] | length(@) at path : JMESPath query failed: Invalid type for: <nil>, expected: []jmespath.JpType{\"string\", \"array\", \"object\"}]"
}