What are the components that need to change for this? Would the profile controller need to create waypoint proxies for each user's namespace?

Waypoint proxies are automatically created when the Gateway resource gets created (so is a bit simpler than crafting deployments/services). You can configure it in such a way that there's 1 instance per namespace that handles all L7 traffic for that namespace.

@peterj one waypoint proxy per dynamic on demand namespace would be a critical change.

That's how the ambient is designed to handle L7. So instead of running sidecar next to every workload, you run 1 waypoint (L7) proxy per namespace.

Yeah I think the profile controller would be instantiating waypoint proxies for each user namespace. That shouldn't be so hard though (the profile controller already creates other per-namespace resources)

This probably gets more complicated wrt to the Gateway API though. Do we need to migrate all VirtualServices and such to the Gateway API too? I'm not clear on what is interoperable

The mixing of resources - VirtualService and Gateway API resources - is not supported in ambient. So, it's either Gateway API + HTTPRoute/TLSroute/TCPRoute or no Gateway API.

Is there an inventory of VirtualServices (and features within) currently used by kubeflow?

From the Istio docs, the following features are supported in HTTPRoutes (Gateway API):

• matching on paths, headers
• mirroring
• weight-based routing
• timeouts

Someone else will have to speak to it, but from what I can tell the features from VirtualServices used now are covered by HTTPRoute. I'm less sure about AuthorizationPolicies and things like authentication on the gateway - do you know the state of those with ambient istio+gateway apis?

The RequestAuthentication and AuthorizationPolicies can be used as-is. The only difference is that you have to use "targetRefs" instead of a selector label to point to the service. Here's an example from the docs:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: view-only
  namespace: default
spec:
 # THIS IS DIFFERENT- you're targeting a Gateway, instead of using labels.
  targetRefs:
  - kind: Gateway
    group: gateway.networking.k8s.io
    name: default
  action: ALLOW
  rules:
  - from:
    - source:
        namespaces: ["default", "istio-system"]
    to:
    - operation:
        methods: ["GET"]

And request auth:

apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
 name: "jwt-example"
 namespace: foo
spec:
 targetRef:
   kind: Gateway
   group: gateway.networking.k8s.io
   name: httpbin-gateway
 jwtRules:
 - issuer: "[email protected]"
   jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.22/security/tools/jwt/samples/jwks.json"

But, of course, it would be great to test these things out beforehand :) Is there a good walkthrough and a collection scenarios one can run through that touch these features? (I am not so familiar with kubeflow, but if someone points me to the scenarios, I could probably test it out with Gateway API)

In a PR you will just trigger many scenarios automatically, so that is the easiest way to test. Just check out all the GitHub workflows

this is great work, thanks for moving this forward.

ty @peterj this is really helpful clarifications!

Do you know how the Gateway API works for setting up authentication at the gateway? I'm thinking of this EnvoyFilter - does it have an equivalent workflow for the new api?

I think there might be an easier way to define the ext-authz that doesn't use EnvoyFile -- https://istio.io/latest/docs/tasks/security/authorization/authz-custom/

I can see the oauth2-proxy is using the configuration above, but I am not sure why the oidc-service is using an EnvoyFilter... It should work with the same configuration I think

Also, to answer the original question - based on the docs here the targetRefs could be used to configure the Envoyfilter (but again, should be tested :))

oidc-authservice will be eliminated soon :-)

This is an amazing effort @peterj! I'd also like to help on the changes required.

Another important component here though is going to be KServe and Knative, that create a lot of VirtualServices under the hood. Knative specifically sounds very scary since we have no influence over, from the Kubeflow side.

cc @yuzisun

EDIT: Maybe the only way to start with this would be with RawDeployment mode of KServe, which doesn't require Istio. But we need to try it out. My concern would be with the Ingress (instead of GW) that needs istio to implement the IngressClass https://kserve.github.io/website/latest/admin/kubernetes_deployment/#1-install-istio

The Ingress resource still works with Istio (i.e. Istio implements the ingress class - ref), but it would make sense to move that over to Kubernetes Gateway API.

I haven't tested anything in knative yet, but they have some discussion of ambient working

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

/lifecycle frozen

Something that concerns me with ambient is that I can't figure out how to do deny-by-default. With sidecar, you can create an allow-nothing policy that forces all communication to need an Authorization Policy enabling it. But with ambient, I don't see a good way to do that.

Does anyone have a solution? It feels like a pretty big hole so I assume I'm missing something here

I'm trying to deploy kubeflow with istio ambient mode but I find that it might be difficult to migrate the ext-auth: istio/istio#51214

I'm trying to deploy kubeflow with istio ambient mode but I find that it might be difficult to migrate the ext-auth: istio/istio#51214

We will try istio-cni first for rootless istio, so please help there as well #2907

KServe community is working on Gateway API migration. See kserve/kserve#3952