Optimize Trivy node-collector memory requirements on Kubernetes

[marcofranssen]

Description

We run on EKS using Karpenter to do the autoscaling. Whenever we do a couple of deployments we always get nodes much larger then required due to the node-collector job that has a relative high memory request. After the job finishes that nodes memory utilization drops back to around 30/50% as many of our workloads are quite memory optimized.

We tried limiting the memory be doing less requests and limits, but that results in OOM errors on larger nodes that run more pods. We have enabled all the scanning features.

It would be great if the node-collector could be fully memory optimized and maybe offload the work to another Pod in the cluster that does the actual scanning. This way cluster memory utilization will be optimized as not every single node has to reserve a large amount of memory for the temporary job to execute. Instead there will be one pod in the cluster that takes the work to process the scan results. This should result in an overal much better memory utilization in the cluster without compromising the security features Trivy offers.

At the moment we are considering disabling the trivy node-collector as the overall memory utilization of our clusters on avarage is below 50%. Basically the more nodes the worse it gets.

Target

Kubernetes

Scanner

None

[chen-keinan]

@marcofranssen thanks for the input, I'll investigate it to under the amount of memory consumption needed

[marcofranssen]

Some additional information. The larger our nodes, the more memory is needed, as we limited the memory resources to prevent worse memory utilization most of the node-collector jobs fails with a OOM error now. Giving it more memory is not an option as it would even further lower the cluster memory utilization.

1. Optimizing the memory usage of the collector is a potential quick win for short term.
2. Redisigning the way nodes are scanned (maybe distributed in a agent-server model) might become a better long term solution. E.g. similar like fluentbit/prometheus.