benchmark

Performance Benchmarking

This guide will help you to take performance measurements in your setup using STUNner. Both running STUNner locally (outside of Kubernetes) and running STUNner in Kubernetes can be evaluated. You can then compare the locally measured result with the result obtained from Kubernetes and figure out performance bottlenecks.

Locally there is no installation needed, it should take less than a minute to make a measurements. If you have a Kubernetes cluster up and running, the installation and measurement should take a few minutes max.

Tools

The tools used in the measurement are the following:

• iperf: Used for creating traffic flows between the clients and the server.
• turncat: Used for opening a connection through STUNner to the iperf server.
• STUNner: Our TURN server exposed to turncat clients.

Measurement Setup

Local setup

All the components are running locally using localhost to simulate the network.

Kubernetes setup

iperf and turncat clients are running locally, both STUNner and iperf server are running inside a Kubernetes Cluster in a pod.

Prerequisites

You must have iperfv2, jq, and Go installed locally to run this tutorial. For Kubernetes benchmarks, you also need a running STUNner installation.

Install locally

You are good to go. No installation steps required.

Install on Kubernetes

Configure STUNner to act as a STUN server towards turncat clients, and to let iperf client's traffic reach the iperf server.

kubectl apply -f iperf-server.yaml
kubectl apply -f performance-stunner.yaml

Executing measurements

Helper script parameters

We bundle a helper script for executing performance measurements. The script uses optional arguments. The flags are the following:

• -h Show help text
• -n Number of turncat clients (more of them can be used, this way each client will forward lesser traffic and none of them becomes the bottleneck while measuring)
• -t Time in seconds to transmit for
• -s Size of the packet in bytes
• -b Bandwidth to send in bits/sec
• -p Platform, can be local or k8s

./benchmark.sh -n 5 -t 5 -s 1000 -b 100000 -p k8s

Performance measuring locally without Kubernetes

The below command will open:

• a stunnerd UDP listener at 127.0.0.1:5001
• one or more turncat clients at 127.0.0.1:90XY (90XY are ports used for measurement purposes starting from 9000) to open a connection through STUNner to the iperf server
• an iperf server at 127.0.0.1:5000
• an iperf client sending its traffic to the turn

An example for:

• a local benchmark
• 5 turncats
• 5 seconds of evaluation time
• 1000 byte packets
• 100 Mbits/sec

./benchmark.sh -n 5 -t 5 -s 1000 -b 100000000 -p local

Performance measuring with Kubernetes

The below command will open:

• one or more turncat clients at 127.0.0.1:90XY (90XY are ports used for measurement purposes starting from 9000) to open a connection through STUNner to the iperf server. Traffic will be forwarded to the STUNner public address obtained from STUNner configuration
• an iperf client sending its traffic to the turn

STUNner and iperf are running inside the Kubernetes Cluster.

An example for:

• a Kubernetes benchmark
• 5 turncats
• 5 seconds of evaluation time
• 1000 byte packets
• 100 Mbits/sec

./benchmark.sh -n 5 -t 5 -s 1000 -b 100000000 -p k8s

Expected output / Getting the results

The output will be a standard iperf server output trimmed to show only the results. There are per-connections results and summarized results.

• Per-connection results: The number of connections are set with the -n argument of the helper script. Rows starting like [ 4] show per-connection results (e.g., throughput, jitter, and latency).
• Summarized results: The [SUM] row summarizes the amount of transferred data, the effective bandwidth, the rate of the dropped/lost packets and the total number of sent packets and finally the packets/second (pps) rate.

Next, we see an example output for a local measurement and a measurement in Kubernetes.

Local measurement

In a local measurement the output contains a single summarized test. You should see a similar output:

Number of concurrent turncat clients: 5
Evaluation time: 1 sec
Packet size: 1000 bytes
Bandwidth: 10000 Kbits/sec or 10 Mbits/sec per turncat client
Platform: local
Results
[ ID] Interval            Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS NetPwr
[  4] 0.0000-0.9977 sec  1.19 MBytes  10.0 Mbits/sec   0.038 ms 3/1253 (0.24%) 0.061/0.020/3.622/0.161 ms 1253 pps 20465
[  1] 0.0000-0.9992 sec  1.19 MBytes  10.0 Mbits/sec   0.049 ms 0/1253 (0%) 0.107/0.021/3.708/0.186 ms 1254 pps 11770
[  3] 0.0000-0.9980 sec  1.19 MBytes  10.0 Mbits/sec   0.003 ms 0/1253 (0%) 0.040/0.023/2.927/0.145 ms 1256 pps 31217
[  2] 0.0000-0.9989 sec  1.19 MBytes  10.0 Mbits/sec   0.042 ms 1/1253 (0.08%) 0.115/0.024/1.391/0.091 ms 1253 pps 10864
[  5] 0.0000-0.9977 sec  1.19 MBytes  10.0 Mbits/sec   0.003 ms 1/1253 (0.08%) 0.036/0.023/1.818/0.097 ms 1255 pps 34840
[ ID] Interval            Transfer     Bandwidth      Write/Err  PPS
[SUM] 0.0000-0.9985 sec  5.97 MBytes  50.2 Mbits/sec  5/6265     6269 pps

Kubernetes measurement

In case of a Kubernetes measurement, the output contains one or more summarized tests. In case the user reruns the script, iperf outputs will be appended. You should see an output similar to this:

Number of concurrent turncat clients: 5
Evaluation time: 1 sec
Packet size: 1000 bytes
Bandwidth: 1000 Kbits/sec or 1 Mbits/sec per turncat client
Platform: k8s
Results
[ ID] Interval            Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS  inP NetPwr
[  3] 0.0000-4.9589 sec   613 KBytes  1.01 Mbits/sec   0.756 ms    0/  628 (0%)  7.780/ 7.094/56.286/ 4.166 ms  127 pps  985 Byte 16.28
[  4] 0.0000-4.9592 sec   613 KBytes  1.01 Mbits/sec   0.623 ms    0/  628 (0%)  7.739/ 7.121/56.342/ 4.182 ms  127 pps  980 Byte 16.36
[  6] 0.0000-4.9579 sec   613 KBytes  1.01 Mbits/sec   0.593 ms    0/  628 (0%)  8.019/ 7.363/57.748/ 4.300 ms  127 pps 1016 Byte 15.80
[  5] 0.0000-4.9577 sec   613 KBytes  1.01 Mbits/sec   0.769 ms    0/  628 (0%)  7.949/ 7.222/58.249/ 4.400 ms  127 pps 1007 Byte 15.94
[  7] 0.0000-4.9600 sec   613 KBytes  1.01 Mbits/sec   1.017 ms    0/  628 (0%)  7.886/ 7.230/55.612/ 4.098 ms  127 pps  998 Byte 16.06
[SUM] 0.0000-4.9643 sec  2.99 MBytes  5.06 Mbits/sec   0.000 ms    0/ 3140 (0%)
[ ID] Interval            Transfer     Bandwidth        Jitter   Lost/Total  Latency avg/min/max/stdev PPS  inP NetPwr
[  8] 0.0000-4.9594 sec   625 KBytes  1.03 Mbits/sec   0.078 ms    0/  628 (0%)  8.837/ 7.591/56.173/ 6.396 ms  129 pps 1.11 KByte 14.60
[  6] 0.0000-4.9435 sec   604 KBytes  1.00 Mbits/sec   0.112 ms    9/  628 (1.4%)  7.554/ 7.313/11.592/ 0.385 ms  125 pps  946 Byte 16.58
[  4] 0.0000-4.9519 sec   605 KBytes  1.00 Mbits/sec   0.091 ms    8/  628 (1.3%)  7.950/ 7.735/11.985/ 0.371 ms  125 pps  995 Byte 15.75
[  3] 0.0000-4.9594 sec   613 KBytes  1.01 Mbits/sec   0.081 ms    0/  628 (0%)  8.339/ 7.792/56.527/ 4.156 ms  127 pps 1.03 KByte 15.18
[  5] 0.0000-4.9441 sec   604 KBytes  1.00 Mbits/sec   0.123 ms    9/  628 (1.4%)  7.538/ 7.320/11.560/ 0.363 ms  125 pps  944 Byte 16.61
[SUM] 0.0000-4.9597 sec  2.98 MBytes  5.04 Mbits/sec   0.000 ms   14/ 3140 (0.45%)
[SUM] 0.0000-4.9597 sec  12 datagrams received out-of-order

Notice that the average packets/second rate will be slightly lower in case of a hosted Kubernetes cluster than in case of a local STUNner installation.

Caveats

• It is advised to repeat the measurement with different packet sizes. Recommended packet sizes in bytes are 64, 128, 256, 512, 1024, and 1200. Small packet sizes result lower effective throughput (when packet drop is < 1%).
• Measuring measuring one-way latency with iperf requires the clocks at the iperf client and server to be synchronized. Without this the results may be corrupted, and you may even see negative latencies.

Help

STUNner development is coordinated in Discord, feel free to join.