Skip to content

Latest commit

 

History

History
267 lines (220 loc) · 17.1 KB

README.md

File metadata and controls

267 lines (220 loc) · 17.1 KB
Raw

Kubeflow Authentication using Oauth2 Proxy

Istio envoyExtAuthzHttp

This is Istio's recommended approach for External Authorization1. It is not limited to the use of oauth2-proxy2 alone. This method is an industry standard, meeting all of Kubeflow's current and foreseeable authentication needs.

Kubeflow Pipelines User and M2M Authentication and Authorization

The Kubeflow Pipelines component relies on the built-in kubernetes functionalities to authenticate and authorize user requests, specifically the TokenReviews3 and SubjectAccessReview4.

The best way to describe how it works is to explain with an example. Lets analyze the flow when a client calls the API to list the KF Pipeline runs:

  1. api-server starts endpoints in:

    https://github.com/kubeflow/pipelines/blob/2.0.5/backend/src/apiserver/main.go#L95

    Focusing on the pipelines run service:

    1. Register Run Service:
    2. proto RPC definition of ListRunsV1
    3. code definition of ListRunsV1
    4. ListRunsV1 calls internal method listRuns
    5. listRuns calls internal method canAccessRun which itself calls s.resourceManager.IsAuthorized
    6. ResourceManager.IsAuthorized first tries to authenticate over every available authenticator, which are the TokenReviewAuthenticator and HTTPHeaderAuthenticator
    7. TokenReviewAuthenticator.GetUserIdentity gets the token from Authorization header and calls the K8s Auth authv1.TokenReview with given token which in return provides userInfo := review.Status.User. GetUserIdentity return userInfo.Username which at this point is the system:serviceaccount:default:default.
    8. Next in ResourceManager.IsAuthorized a SubjectAccessReview is created with r.subjectAccessReviewClient.Create with arguments specifying RBAC verbs provided in code definition of RunServer.listRuns. If the user (sa) is not authorized, an error is thrown

  2. User calls api to list pipeline runs as unauthorized service account.

    • This can be done by running Pod with curl in default namespace: 
      $ kubectl -n default run -ti --rm curl --image curlimages/curl --command -- sh
# v1beta1
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{"error":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list v1beta1 runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}
# v2beta1
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{"error":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","code":7,"message":"Failed to list runs: Failed to list runs due to authorization error. Check if you have permission to access namespace kubeflow-user-example-com: Failed to access run . Check if you have access to namespace kubeflow-user-example-com: PermissionDenied: User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,}): Unauthorized access","details":[{"@type":"type.googleapis.com/google.rpc.Status","code":7,"message":"User 'system:serviceaccount:default:default' is not authorized with reason:  (request: \u0026ResourceAttributes{Namespace:kubeflow-user-example-com,Verb:list,Group:pipelines.kubeflow.org,Version:v1beta1,Resource:runs,Subresource:,Name:,})"}]}

  3. User calls api to list pipeline runs as authorized service account.

    • This can be done by running Pod with curl in kubeflow-user-example-com namespace specifying correct service account: 
      $ kubectl -n kubeflow-user-example-com run -ti --rm curl --image curlimages/curl --command --overrides='{"spec": {"serviceAccountName": "default-editor"}}' -- sh
# v1beta1
~ $ curl "istio-ingressgateway.istio-system/pipeline/apis/v1beta1/runs?resource_reference_key.type=NAMESPACE&resource_reference_key.id=kubeflow-user-example-com" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{}  # empty response which is fine because no pipeline runs exist
# v2beta1
~ $ curl istio-ingressgateway.istio-system/pipeline/apis/v2beta1/runs?namespace=kubeflow-user-example-com -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)"
{}  # empty response which is fine because no pipeline runs exist

Authentication and Authorization analysis diagram for Kubeflow Pipelines

Kubeflow Auth Diagram

Change the default authentication from "Dex + Oauth2-proxy" to "Oauth2-proxy" only

The authentication in Kubeflow evolved over time and we dropped envoyfilters and oidc-authservice in favor of RequestAuthentication and Oauth2-proxy in Kubeflow 1.9. auth-flow

You can adjust OAuth2 Proxy to directly connect to your own IDP(Identity Provider) suchg as GCP, AWS, Azure etc:

  1. Create an application on your IdP (purple line)
  2. Change your OAuth2 Proxy issuer to your IdP. Of course never ever directly, but with kustomize overlays and components.

Here is an example of patching oauth2-proxy to connect directly to Azure IDP and skip Dex. This is enterprise integration so feel free to hire consultants or pay for commercial distributions if you need more help. For example Azure returns rather large headers compared to other IDPs, so maybe you need to annotate the nginx-ingress to support that.

# based on https://github.com/kubeflow/manifests/blob/master/common/oauth2-proxy/base/oauth2_proxy.cfg
# and https://oauth2-proxy.github.io/oauth2-proxy/configuration/providers/azure/
apiVersion: v1
kind: ConfigMap
metadata:
  name: oauth2-proxy
  namespace: oauth2-proxy
data:
  oauth2_proxy.cfg: |
    provider = "oidc"
    oidc_issuer_url = "https://login.microsoftonline.com/$MY_TENANT/v2.0"
    scope = "openid email profile" # remove groups since they are not used yet
    email_domains = [ "*" ]

    # serve a static HTTP 200 upstream on for authentication success
    # we are using oauth2-proxy as an ExtAuthz to "check" each request, not pass it on
    upstreams = [ "static://200" ]

    # skip authentication for these paths
    skip_auth_routes = [
      "^/dex/",
    ]

    # requests to paths matching these regex patterns will receive a 401 Unauthorized response
    # when not authenticated, instead of being redirected to the login page with a 302,
    # this prevents background requests being redirected to the login page,
    # and the accumulation of CSRF cookies
    api_routes = [
      # Generic
      # NOTE: included because most background requests contain these paths
      "/api/",
      "/apis/",

      # Kubeflow Pipelines
      # NOTE: included because KFP UI makes MANY background requests to these paths but because they are
      #       not `application/json` requests, oauth2-proxy will redirect them to the login page
      "^/ml_metadata",
    ]

    skip_provider_button = true
    set_authorization_header = true
    set_xauthrequest = true
    cookie_name = "oauth2_proxy_kubeflow"
    cookie_expire = "24h"
    cookie_refresh = "1h" # This improves the user experience a lot
    redirect_url = "https://$MY_PUBLIC_KUBEFLOW_DOMAIN/oauth2/callback"
    relative_redirect_url = false
  1. In the istio-system namespace is a RequestAuthentication resource. You need to change its issuer to your own IdP, or even better create an additional one.
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: azure-aad-requestauthentication
  namespace: istio-system
spec:
  # we only apply to the ingress-gateway because:
  #  - there is no need to verify the same tokens at each sidecar
  #  - having no selector will apply to the RequestAuthentication to ALL
  #    Pods in the mesh, even ones which are not part of Kubeflow
  selector:
    matchLabels:
      app: istio-ingressgateway

  jwtRules:
  - issuer: https://login.microsoftonline.com/$MY_TENANT/v2.0

    # `forwardOriginalToken` is not strictly required to be true.
    # there are pros and cons to each value:
    #  - true: the original token is forwarded to the destination service
    #          which raises the risk of the token leaking
    #  - false: the original token is stripped from the request
    #           which will prevent the destination service from
    #           verifying the token (possibly with its own RequestAuthentication)
    forwardOriginalToken: true

    # This will unpack the JWTs issued by Dex or other IDPs into the expected headers.
    # It is applied to BOTH the m2m tokens from outside the cluster (which skip
    # oauth2-proxy because they already have a dex JWT), AND user requests which were
    # authenticated by oauth2-proxy (which injected a dex JWT).
    outputClaimToHeaders:
    - header: kubeflow-userid
      claim: email
    - header: kubeflow-groups
      claim: groups

    # We explicitly set `fromHeaders` to ensure that the JWT is only extracted from the `Authorization` header.
    # This is because we exclude requests that have an `Authorization` header from oauth2-proxy.
    fromHeaders:
    - name: Authorization
      prefix: "Bearer "

You can also add more RequestAuthentication to support other issuers as for example for M2M access from github actions as explained in the root level Readme.md. This feature is useful when you need to integrate Kubeflow with your current CI/CD platform (GitHub Actions, Jenkins) via machine-to-machine authentication. The following is an example for obtaining and using a JWT token From your IDP with Python, but you can also just take a look at our CI/CD test that uses simple Kubernetes serviceaccount tokens to access KFP, Jupyterlabs etc. from GitHub Actions.

import requests
token_url = "https://your-idp.com/oauth/token"
client_id = "YOUR_CLIENT_ID"
client_secret = "YOUR_CLIENT_SECRET"
username = "YOUR_USERNAME"
password = "YOUR_PASSWORD"
# request header
headers = {
    "Content-Type": "application/x-www-form-urlencoded"
}
data = {
    "grant_type": "password",
    "client_id": client_id,
    "client_secret": client_secret,
    "username": username,
    "password": password,
    "scope": "openid profile email"  #change your scope
}
response = requests.post(token_url, headers=headers, data=data)
TOKEN = response.json()['access_token']

import kfp
kubeflow_host="https://your_host"
pipeline_host = kubeflow_host + "/pipeline" 
client = kfp.Client(host=pipeline_host, existing_token=TOKEN)
print(client.list_runs(namespace="your-profile-name"))

Kubeflow Notebooks User and M2M Authentication and Authorization

The underlying mechanism is the same as in Kubeflow Pipelines.

Similarly, to explain how it works, let's analyze the code step by step, starting from the api route definition for listing notebooks:

KServe Authentication

The analysis of KServe auth capabilities suggests that while it's possible to limit access to only authenticated agents, there might be some improvements required to enable access only to authorized agents.

This is based on the following:

  1. KServe Controller Manager patch integrating kube-rbac-proxy5.

    This suggests the kserve might use the same mechanism based on SubjectAccessReviews. Having a look at the kubeflow/manifests I see it's not enabled.

  2. Search through the docs and code:

    The docs above mention that while it's possible to enable authentication, authorization is more complicated and probably we need to add AuthorizationPolicy...

    create an Istio AuthorizationPolicy to grant access to the pods or disable it

    Most probably some work is needed to enable authorized access to kserve models.

Links

Footnotes

  1. External Authorization ↩

  2. oauth2-proxy ↩

  3. Kubernetes TokenReview ↩

  4. Kubernetes SubjectAccessReview ↩

  5. Kube RBAC Proxy ↩