More comprehensive securityContext settings in helm chart

[rptaylor]

In #874 , customizable securityContexts were added for the pod, and the manager container, but not the kube-rbac-proxy container.

Consequently it is not possible to make the kueue chart abide by Pod Security Standards and it can not be installed on a cluster that enforces them.
Rather than introducing more customizability for another container, it should just define the minimal securityContext that it requires. What does kube-rbac-proxy container do? Will it work with dropping all capabilities by default, RuntimeDefault seccompprofile, allowPrivilegeEscalation: false ?

I can make a PR but the security requirements needed for the application to run properly must be defined (e.g. by someone who knows the code base and what the application does).

[rptaylor]

It looks like this component will be removed: #3760

[kannon92]

I think adding this would be useful.

@mbobrovskyi and @mimowo i don’t think the rbac proxy will go into 0.10 so should we implement some support for this?

[mimowo]

Actually, maybe I should prioritize that PR. I would prefer to invest effort into the container that is going away soon-ish

[mimowo]

@rptaylor @kannon92 the change #3760 was a little bit too big to include for 0.10, but it solves the concern of the container long-term for 0.11. So I would suggest we close the issue.

For the short-term work-around I think you could patch the Kueue deployment for 0.10 setting the appropriate security context - TBH I'm not sure what is the optimal setting here.

[kannon92]

Sounds good to me.

@rptaylor kueue does not manage the proxy image so we aren’t sure the impacts of the security settings. If you find out what works please feel free to add a doc update or update this thread with your findings.

[rptaylor]

Thanks for finishing that PR! Do you know when v0.11 will be released?

[mimowo]

Generally Kueue is aiming for releases every 2-3 months. So, the next we are planning for Feb 2025, but it might slip to March.

cc @mwysokin @mwielgus

[rptaylor]

Actually this also pertains to the manager container. If PSS is enforced on a cluster, these additional settings are required to install kueue, and can be set with .Values.controllerManager.manager.podSecurityContext and/or .Values.controllerManager.manager.containerSecurityContext thanks to #878

          capabilities:
            drop:
            - ALL
          seccompProfile:
            type: RuntimeDefault

But if it can be confirmed that the manager container doesn't need any capabilities or special seccomp profile, wouldn't it be better to make kueue more secure by default and reduce the extra steps needed to use PSS?

[mimowo]

But if it can be confirmed that the manager container doesn't need any capabilities or special seccomp profile

I would be surprised if kueue manager requires a special seccomp profile

wouldn't it be better to make kueue more secure by default and reduce the extra steps needed to use PSS?

I'm open to the possibility, but can you explain more precisely what you mean? What are the steps you want to reduce and how? changing the default?

If this continues to work on kind, gke, openshift etc. ootb, and we keep it configurable for some more restrictive environments, then I'm good to try. cc @dgrove-oss

[rptaylor]

@mimowo Yes, if it is confirmed that kueue does not require any special capabilities or seccomp profile, then I would propose to drop all capabilities and set RuntimeDefault by default, as in #3850 (comment)

[mimowo]

I see, I did a sanity check on GKE and it worked fine with the snippet proposed in #3850 (comment).

So, I'm good to move forward with that, but we need to be careful about current installations, so maybe we should consider an opt-out configuration parameter in help, wdyt @tenzen-y ? @rptaylor do you see some risks, where would you be coming from?

Feel free to post a PR.