Kubernetes intra node TCP higher throughput than UDP

[guyversteeg]

Context

• Version of iperf3: 3.9

• Hardware: x86_64

• Operating system (and distribution, if any): Ubuntu 21.10

• Kubernetes: 1.23

• CNI: Calico, Cilium, Antrea

Bug Report

• Expected Behavior
  TCP Traffic has a throughput of around 25-30Gbit and multiple tests can be run simultaneously intra node while keeping this amount of throughput. I expect UDP traffic to have a throughput as high or higher than TCP traffic.

• Actual Behavior
  UDP throughput is around 10% of TCP throughput.

• Steps to Reproduce

• TCP behavior:
  iperf3 -c 192.168.104.3 -b 0

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 29.4 GBytes 25.2 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 29.4 GBytes 25.2 Gbits/sec receiver

UDP behavior:
iperf3 -c 192.168.104.3 -u -b 0

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 2.34 GBytes 2.01 Gbits/sec 0.000 ms 0/1762940 (0%) sender
[ 5] 0.00-10.00 sec 2.34 GBytes 2.01 Gbits/sec 0.001 ms 6553/1762940 (0.37%) receiver

iperf3 -c 192.168.104.3 -u -b 0/50

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 2.50 GBytes 2.15 Gbits/sec 0.000 ms 0/1879200 (0%) sender
[ 5] 0.00-10.00 sec 2.49 GBytes 2.14 Gbits/sec 0.001 ms 8486/1879200 (0.45%) receiver

iperf3 -c 192.168.104.3 -u -b 0 -l 50000

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 8.35 GBytes 7.18 Gbits/sec 0.000 ms 0/179390 (0%) sender
[ 5] 0.00-10.00 sec 8.31 GBytes 7.14 Gbits/sec 0.002 ms 1007/179390 (0.56%) receiver

Are these results the expected behaviour or is there something I'm still missing?

[swlars]

Have you tried using the -w option to increase the socket buffer size? Maybe try 64K and increase it from there.

[davidBar-On]

Also, TCP default packet size is 128KB and UDP default packet size is about 1.5KB. To compare apples to apples both should be the same. Try for example testing both TCP and UDP using -l 32K.

[guyversteeg]

Thank you for your replies.

When using -l 32K for both UDP and TCP there remains a difference in throughput:
iperf3 -c 172.12.1.6 -l 32K

[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 21.9 GBytes 18.8 Gbits/sec 75 sender
[ 5] 0.00-10.00 sec 21.9 GBytes 18.8 Gbits/sec receiver

iperf3 -c 172.12.1.6 -u -b 0 -l 32K

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 7.42 GBytes 6.38 Gbits/sec 0.000 ms 0/243300 (0%) sender
[ 5] 0.00-10.00 sec 7.40 GBytes 6.36 Gbits/sec 0.001 ms 851/243300 (0.35%) receiver

Increasing the socket buffer size does not have a much effect on the throughput as far as I can tell:
iperf3 -c 172.12.1.6 -u -b 0

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 4.04 GBytes 3.47 Gbits/sec 0.000 ms 0/3100820 (0%) sender
[ 5] 0.00-10.00 sec 4.00 GBytes 3.44 Gbits/sec 0.001 ms 27330/3100820 (0.88%) receiver

iperf3 -c 172.12.1.6 -u -b 0 -w 128K

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams
[ 5] 0.00-10.00 sec 3.99 GBytes 3.43 Gbits/sec 0.000 ms 0/3065180 (0%) sender
[ 5] 0.00-10.00 sec 3.95 GBytes 3.39 Gbits/sec 0.002 ms 34034/3065180 (1.1%) receiver

[davidBar-On]

It seems that the UDP throughput is in the sending side (the client). This may be because of limited size of the UDP buffers on the client side. This is assuming that this is not a CPU issue, i.e. that the TCP stack performance is not much better that the UDP stack (and that TCP is not using more processors than UDP).

If UDP buffers are the issue, then maybe TCP buffers need to be tuned. For example, see here and UDP Tuning here.

One more thing that may help is to limit the number of UDP writes, by limiting the requested bandwidth. That may reduced the overhead of failed UDP packets writes and therefore increase performance. E.g. try iperf3 -c 172.12.1.6 -u -l 32K -b 8G (or iperf3 -c 172.12.1.6 -u -l 32K -b 8G/10). If the required bitrate is achieved, you can try increasing the requested bit rate and see where it gets to.