Promote spark from contribution to application

[juliusvonkohout]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

master

Describe your issue

@rimolive do you remember who wanted to help with the spark operator? I think the main problem is that

1. it does not work with istio "sidecar.istio.io/inject": "true" https://github.com/kubeflow/manifests/blob/73cbecfe604e84dfc7c0851630a6eb5733022dea/contrib/spark/sparkapplication_example.yaml#L24C7-L24C41

2. We are missing a securitycontext in https://github.com/kubeflow/manifests/blob/73cbecfe604e84dfc7c0851630a6eb5733022dea/contrib/spark/sparkapplication_example.yaml and it should be upstreamed to the spark repository

    securityContext:
      capabilities:
        drop:
          - ALL
      runAsUser: 185
      runAsGroup: 0
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      seccompProfile:
        type: RuntimeDefault

3. We need to add runAsGroup:0 or 185 in the securitycontext of https://github.com/kubeflow/manifests/blob/master/contrib/spark/spark-operator/base/kustomization.yaml and we need to upstream it

4. We need a synchronization script in /hack derived from the current makefile

5. upgrade to the latest version

[juliusvonkohout]

CC @GezimSejdiu

[juliusvonkohout]

There is some security progress in https://github.com/kubeflow/spark-operator/releases/tag/v2.1.0

[jacobsalway]

1. Could you go into more detail on the issues you're seeing with Istio? I don't think Istio is needed for either the operator controller/webhook or for RPC/network shuffle within the Spark application. If Istio ingress gateways are being used for other components in Kubeflow, maybe there's a future enhancement to support this for the Spark UI as the current integration is intended for NGINX ingress.
2. Do you mean the examples in our repository should have this securityContext?
3. I'll raise a PR to upstream this but I think the only change needed is the seccompProfile. We also support readOnlyRootFilesystem: true.
4. Is this a script for your or our repository?

[juliusvonkohout]

1. Could you go into more detail on the issues you're seeing with Istio? I don't think Istio is needed for either the operator controller/webhook or for RPC/network shuffle within the Spark application. If Istio ingress gateways are being used for other components in Kubeflow, maybe there's a future enhancement to support this for the Spark UI as the current integration is intended for NGINX ingress.

Istio support for the application in the user namespaces is the problem right now. By default the sparkcluster application must support the sidecars that are injected automatically and we must be on the servicemesh. Right now we are destroying the integration and security

manifests/contrib/spark/sparkapplication_example.yaml

Line 18 in 2ebc576

"sidecar.istio.io/inject": "false"

since I did not have time to test it with proper Istio support.

2. Do you mean [the examples in our repository](https://github.com/kubeflow/spark-operator/tree/master/examples) should have this `securityContext`?

Yes kubernetes best practices is that the operator, webhook and sparkapplication have a PSS restricted securitycontext https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted

3. I'll raise a PR to upstream this but I think the only change needed is the `seccompProfile`. We also support `readOnlyRootFilesystem: true`.

Yes readOnlyRootFilesystem: true can make sense for the operator and webhook

4. Is this a script for your or our repository?

Yes this is a script that we have to add here https://github.com/kubeflow/manifests/tree/master/hack analogously to the other scripts.