Questions about Pod Security Admission

[jlowe64]

Hello,

I am very new to both kubernetes and running Talos linux. So apologies for the very basic questions, but I have basically narrowed my issues down to how talos linux sets up kubernetes.

I'm in need of a workflow that works for deploying things onto kubernetes without tons of manual work involved in editing the machine configs and predicting which namespaces things will go into. I still wanna do things the "right way" but I also want to balance that with being able to get beyond the boilerplate of managing the actual cluster, and move on to deployments.

My issue comes when I try to deploy helm charts or applications onto the cluster, I always run into a whole list of errors and while I sorta understand what is causing the issues (it's the pod security admission policy) and I know I should tag the namespace with whatever policy I want, but even if I tag (which I did in this case with a more privileged policy) I still end up with the following style of errors.

I'm more or less looking for advice on a workflow that you might use to manage this situation without tons of overhead.

W0311 22:10:03.008586   25369 warnings.go:70] would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true), hostPort (container "agent-ha-node" uses hostPort 50053), privileged (container "agent-ha-node" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "agent-cluster-grpc-probe", "agent-ha-node" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "agent-cluster-grpc-probe", "agent-ha-node" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "device", "sys", "run-udev", "plugin-dir" use restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "agent-cluster-grpc-probe", "agent-ha-node" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "agent-cluster-grpc-probe", "agent-ha-node" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.062174   25369 warnings.go:70] would violate PodSecurity "restricted:latest": restricted volume types (volumes "run", "containers", "pods" use restricted volume type "hostPath"), runAsNonRoot != true (pod or container "promtail" must set securityContext.runAsNonRoot=true), runAsUser=0 (pod must not set runAsUser=0), seccompProfile (pod or container "promtail" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.062178   25369 warnings.go:70] would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true), hostPort (container "csi-driver-registrar" uses hostPort 10199), privileged (container "csi-node" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "csi-node", "csi-driver-registrar" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "csi-node", "csi-driver-registrar" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "device", "sys", "run-udev", "registration-dir", "plugin-dir", "kubelet-dir" use restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "csi-node", "csi-driver-registrar" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "csi-node", "csi-driver-registrar" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.062357   25369 warnings.go:70] would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true), hostPort (containers "metrics-exporter-pool", "io-engine" use hostPorts 10124, 9502), privileged (container "io-engine" must not set securityContext.privileged=true), allowPrivilegeEscalation != false (containers "agent-core-grpc-probe", "etcd-probe", "metrics-exporter-pool", "io-engine" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "agent-core-grpc-probe", "etcd-probe", "metrics-exporter-pool", "io-engine" must set securityContext.capabilities.drop=["ALL"]), restricted volume types (volumes "device", "udev", "configlocation" use restricted volume type "hostPath"), runAsNonRoot != true (pod or containers "agent-core-grpc-probe", "etcd-probe", "metrics-exporter-pool", "io-engine" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "agent-core-grpc-probe", "etcd-probe", "metrics-exporter-pool", "io-engine" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.115848   25369 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "obs-callhome" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "obs-callhome" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "obs-callhome" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "obs-callhome" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.115910   25369 warnings.go:70] would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true), allowPrivilegeEscalation != false (containers "api-rest-probe", "csi-provisioner", "csi-attacher", "csi-controller" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "api-rest-probe", "csi-provisioner", "csi-attacher", "csi-controller" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "api-rest-probe", "csi-provisioner", "csi-attacher", "csi-controller" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "api-rest-probe", "csi-provisioner", "csi-attacher", "csi-controller" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.115848   25369 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "agent-core-grpc-probe", "etcd-probe", "api-rest" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "agent-core-grpc-probe", "etcd-probe", "api-rest" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "agent-core-grpc-probe", "etcd-probe", "api-rest" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "agent-core-grpc-probe", "etcd-probe", "api-rest" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.116025   25369 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "etcd-probe", "agent-core", "agent-ha-cluster" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "etcd-probe", "agent-core", "agent-ha-cluster" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "etcd-probe", "agent-core", "agent-ha-cluster" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "etcd-probe", "agent-core", "agent-ha-cluster" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.115950   25369 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "agent-core-grpc-probe", "etcd-probe", "operator-diskpool" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "agent-core-grpc-probe", "etcd-probe", "operator-diskpool" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or containers "agent-core-grpc-probe", "etcd-probe", "operator-diskpool" must set securityContext.runAsNonRoot=true), seccompProfile (pod or containers "agent-core-grpc-probe", "etcd-probe", "operator-diskpool" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.139396   25369 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (containers "volume-permissions", "loki" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "volume-permissions", "loki" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod must not set securityContext.runAsNonRoot=false), runAsUser=0 (container "volume-permissions" must not set runAsUser=0), seccompProfile (pod or containers "volume-permissions", "loki" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
W0311 22:10:03.146539   25369 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "volume-permissions" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (containers "volume-permissions", "etcd" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "volume-permissions" must set securityContext.runAsNonRoot=true), runAsUser=0 (container "volume-permissions" must not set runAsUser=0), seccompProfile (pod or containers "volume-permissions", "etcd" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
NAME: mayastor
LAST DEPLOYED: Sat Mar 11 22:09:58 2023
NAMESPACE: mayastor
STATUS: deployed
REVISION: 1
NOTES:
OpenEBS Mayastor has been installed. Check its status by running:
$ kubectl get pods -n mayastor```

[smira]

What you see is not an error, but rather a warning.

Please look into Talos documentation which also references Kubernetes documentaiton.

You can update the profiles enforced/autdited/warned in the machine config if you want to change the defaults.

[jlowe64]

This got me on the right path, but I had already read the documentation. I realised after this that I was missing some items due to getting bogged down with pod security admission defaults. I've changed the defaults to be more permissive and instead to enforce security by default on the namespaces I can control with deployments (without editing everyone's helm charts to tag their own security needs).

Very helpful though!

[SebTardif]

@simira I have a similar issue. By default, Istio appears to not be working... The first Istio official sample fail, so when running: https://istio.io/latest/docs/tasks/traffic-management/ingress/ingress-control/

I get in events: Error creating: pods "httpbin-ff5c59f7c-nw9qb" is forbidden: violates PodSecurity "baseline:latest": non-default capabilities (container "istio-init" must not include "NET_ADMIN", "NET_RAW" in securityContext.capabilities.add)

Then, trying to fix this, it appears I need to install -> https://artifacthub.io/packages/helm/istio-official/cni
But that install didn't appear to be enough to fix this.

However... the parameters has tendency to change for different flavor of Kubernetes, so I wonder if you could take a look of what should be the right parameters, and add Istio in the smoke tests of Talos.

See https://istio.io/latest/docs/setup/additional-setup/cni/#hosted-kubernetes-settings

[smira]

Lower-level things need elevated privileges, so it's expected you mark their namespaces as privileged. They should do that in their Helm charts though.

E.g. look how Flannel does it right:

https://github.com/flannel-io/flannel/blob/df6b8a68762412a8bec1e117994c1730ff608574/Documentation/kube-flannel.yml#L7

[jlowe64]

Since I'm not dealing with a production workload, and it's just a homelab I'll learn on, I've personally flipped the security requirements. Most of the helm charts that other providers are putting out do not tag their namespaces at all with the security needs they require, so I end up with a bunch of totally blocked pods instead. So I've instead changed the default behavior to be privileged and instead I'll enforce baseline on my own non-cluster level deployments. Maybe in the next few months we'll see folks helm charts not require so much overhead to deploy under the pod security admission era.

I might be totally wrong in doing it this way, but it just seemed like a good balance for homelab purposes.

[SebTardif]

@smira The issue is not what you are thinking about. When I deploy that Istio sample app, called httpbin, it's blocked to be deployed not because the httpbin container itself, but because the extra "initialize" container injected by Istio, called istio-init, is blocked. Of course, I could disable security on ALL namespaces where I deploy anything...but it's not what I'm looking for.

Istio know about that issue, that istio-init, which run on the same namespace that the 'business logic' container has integration issues with Pod Security Admission. So they rewrite the workflow, and replaced where that logic run (fixing iptable and maybe more) using that Istio CNI plugin.

So, my issue, is still about getting that Istio CNI plugin to work. Since the Istio doc mention that some parameters may have to be adapted for each flavor of Kubernetes implementation, it's why I have reached to Siderolabs. In my work, Istio is everything, you guys could try this pretty fast, it's just couple of Helm chart install with ALL default installation to see it failing...

Maybe it will happen only with your 'default' installation of flannel -> flannel-io/flannel#1259

Flannel doesn't have network policy, so I'm also wondering why it's the default!? I think I just saw Calico ever been used.

I guess that give me two reason to stay way from Flannel and try something else.

[smira]

I'm not sure what is the question, if Istio is not compatible with Pod Security Admission, you can disable it completely in Talos.

Same way you can use any CNI with Talos, Flannel is just the default provided, and it doesn't cover all usecases.