SSL enabled domain gives: RBAC access denied

[khalidkhushal]

Validation Checklist

• Is this a Kubeflow issue?
• Are you posting in the right repository ?
• Did you follow the Kubeflow installation guideline ?
• Is the issue report properly structured and detailed with version numbers?
• Is this for Kubeflow development ?
• Would you like to work on this issue?
• You can join the CNCF Slack and access our meetings at the Kubeflow Community website. Our channel on the CNCF Slack is here #kubeflow-platform.

Version

1.9

Describe your issue

Hello guys,
I deployed kubeflow v1.9.1-rc.3 on on-premise servers using k3s.
I have added ssl for istio and certs are working fine, but I am facing an issue while accessing dashbaord
Error:
url: https://domain.com/oauth2/start?rd=%2F

error message: upstream connect error or disconnect/reset before headers. reset reason: connection termination

Visrtual Service for https:

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: kubeflow-https
  namespace: istio-system
spec:
  hosts:
    - 'domain.com'  
  gateways:
    - istio-ingressgateway  
  tls:
    - match:
        - sniHosts:
            - 'domain.com'  
      route:
        - destination:
            host: istio-ingressgateway  
            port:
              number: 443 
  http:
    - match:
        - uri:
            prefix: /
      route:
        - destination:
            host: istio-ingressgateway 
            port:
              number: 80   

Please guide on this what I am doing wrong here.

If I remove this Virtualservice

• Http traffic works
• Https fails with: RBAC access denied

Thanks in advance.

Steps to reproduce the issue

• Enable SSL on istio -ingressgateway
• Try to access with https://domain

Put here any screenshots or videos (optional)

No response

[juliusvonkohout]

Hello, the easiest way is to just put a Kubernetes ingress in front of the istio-ingressgateway service and terminate TLS there. If you do not want that you can add the information and certificates directly to istio-ingressgateway service and change it to type loadbalancer. Why do you want to modify the virtual service in the first place?

Check whether you can add the certificates to kubeflow-gateway.

And after all you should document it here such that other users will benefit as well.

[khalidkhushal]

Hi @juliusvonkohout Thank you for responding on this.
I am not changing virtual service, just added a new one that I mentioned in issue description.
But, If I don't create a virtual service I simply get "RBAC: Access Denied" Error while accessing domain with https://.
I have added tls info in istio-ingressgateway as well.

[khalidkhushal]

Done @juliusvonkohout , Thanks.
SSL is working now! 🙌

Adding the steps here if anybody needs it may help.

[khalidkhushal]

Steps to Enable SSL on kubeflow/manifests by placing an Ingress before Istio Gateway:

• We already have cert-manager in kubeflow.
• Add cluster issuer for certificates, in my case I used lets-encrypt. See config below:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: {{ .Values.email }}
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
          ingress:
               class: nginx

• Install Ingress-Nginx to cluster
• Change the ingress-controller service type from NodePort to Loadbalancer
• Add an ingress to your cluster that accepts connection at "/" and transfers the rest of the routing to istio-ingressgateway. See the example config below:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubeflow-ingress
  namespace: istio-system
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    kubernetes.io/ingress.class: "nginx" 
spec:
  tls:
  - hosts:
    - <domain.com>
    secretName: <tls-secret-name>
  rules:
  - host:  <domain.com>
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: istio-ingressgateway
            port:
              number: 80

• Make sure your domain resolves to the external IP in loadbalancer.
• Test SSL, it should be working now.