HTTPRoute Weight Not Honored for Pod-to-Pod Communication via ClusterIP Service

[Manikkumar1988]

What is the issue?

In minikube with latest linkerd installed,

Expectation is
HTTPRoute - should honour the weightage when routing the traffic to desired service for the communication happening through ClusterIP type Service.

Actual is
HTTPRoute - doesn't honour the weightage when routing the traffic to desired service. Its alway directing to the non-desired service. Even when weightage of desired service has 100% while the other has 0%

How can it be reproduced?

1. Create a Namespace and Label it for Linkerd Injection

kubectl create namespace test-app
kubectl annotate namespace test-app linkerd.io/inject=enabled

2. Create two deployments and expose it by service of clusterIP type.
   sample-app-v1.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-v1
  namespace: test-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-app
      version: v1
  template:
    metadata:
      labels:
        app: sample-app
        version: v1
    spec:
      containers:
        - name: http-app
          image: hashicorp/http-echo:latest
          args:
            - "-text=Hello from v1"
          ports:
            - containerPort: 5678

---
apiVersion: v1
kind: Service
metadata:
  name: sample-app-v1
  namespace: test-app
spec:
  selector:
    app: sample-app
    version: v1
  ports:
    - port: 80
      targetPort: 5678

sample-app-v2.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-v2
  namespace: test-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-app
      version: v2
  template:
    metadata:
      labels:
        app: sample-app
        version: v2
    spec:
      containers:
        - name: http-app
          image: hashicorp/http-echo:latest
          args:
            - "-text=Hello from v2"
          ports:
            - containerPort: 5678

---
apiVersion: v1
kind: Service
metadata:
  name: sample-app-v2
  namespace: test-app
spec:
  selector:
    app: sample-app
    version: v2
  ports:
    - port: 80
      targetPort: 5678

3. Create one HTTPRoute with two backends(v2 service & v1 service) and one parentRef(v1 service) and set value of weight for the v2 service as 100%.
   route.yaml

apiVersion: policy.linkerd.io/v1beta2
kind: HTTPRoute
metadata:
  name: bb-route
  namespace: test-app
spec:
  parentRefs:
    - name: sample-app-v1
      kind: Service
      group: core
      port: 80
  rules:
    - backendRefs:
      - name: sample-app-v1
        port: 80
        weight: 0
      - name: sample-app-v2
        port: 80
        weight: 100

4. Run a nginx pod
   kubectl run my-pod --image=nginx -n test-app

5. Run curl request to v1 service.
   kubectl exec -it -n test-app my-pod -c my-pod -- curl -s -o - sample-app-v1

Expected:
Hello from v2

Actual:
Hello from v1

Logs, error output, etc

Expected:
Hello from v2

Actual:
Hello from v1

output of linkerd check -o short

Environment

Kubernetes Version:
Client Version: v1.30.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.0

Cluster Version:
minikube version: v1.33.1
commit: 5883c09216182566a63dff4c326a6fc9ed2982ff

Host OS:
Mac Sonoma 14.5

Linkerd Version:
Client version: edge-24.10.4
Server version: edge-24.10.4

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

maybe

[kflynn]

Hey @Manikkumar1988! Can you show us the status on that HTTPRoute? kubectl get -n test-app httproute bb-route -o yaml? If you're finding that an HTTPRoute or GRPCRoute isn't working, that's always the first thing to check...

[Manikkumar1988]

Thanks for the response, I have mentioned wrong port in the HTTPRoute thats why it wasn't working.
However in my actual, dev env cluster with buoyant image, still issue is reproducible.

[olix0r]

I'm unable to reproduce this:

:; kubectl exec -it -n test-app my-pod -c my-pod -- curl -vs sample-app-v1
*   Trying 10.43.101.1:80...
* Connected to sample-app-v1 (10.43.101.1) port 80 (#0)
> GET / HTTP/1.1
> Host: sample-app-v1
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< x-app-name: http-echo
< x-app-version: 1.0.0
< date: Fri, 15 Nov 2024 20:42:07 GMT
< content-length: 14
< content-type: text/plain; charset=utf-8
< 
Hello from v2
* Connection #0 to host sample-app-v1 left intact

As mentioned, if you encounter this in the future, please capture the route status as well as the output of linkerd diagnostics policy -n test-app svc/sample-app-v1 80.

[olix0r]

We believe this issue may was fixed by #12635, which was included in edge-24.5.4 (released in May).

Your report indicates that you were using a recent edge release, though:

Linkerd Version:
Client version: edge-24.10.4
Server version: edge-24.10.4

If you can reproduce this on a recent Linkerd release, we'd be curious to learn more.