Hit issues open AF_PACKET socket against veth nic

[hanlins]

Hi nff-go community, I was trying to use nff-go in my project. What I want to achieve is to check the original packets' five-tuple and encapsulate the original packets (in IPIP) and set the destination accordingly and send it out through the same interface. I'm trying to use AF_PACKET before moving to XDP/DPDK ports. I hit several issues:

The first issue is when I was trying to use SetReceiverOS and SetSenderOS as the input/output flow function, the process just stuck there and perform any packet processing. I was trying to send the packets through a veth pair interface. Here's the code snippet:

package main

import (
	"flag"
	"fmt"
	"net"

	"github.com/intel-go/nff-go/common"
	"github.com/intel-go/nff-go/flow"
	"github.com/intel-go/nff-go/packet"

	"github.com/hanlins/go-lb/cmd/forwarder/ipip"
)

var (
	src *string
	dst *string
)

func main() {
	device := flag.String("dev", "lo", "network device name")
	src = flag.String("src", "0.0.0.0", "outer IP header src")
	dst = flag.String("dst", "0.0.0.0", "outer IP header dst")
	flag.Parse()

	config := flow.Config{
		NeedKNI:  true,
		DPDKArgs: []string{"--no-pci", "--vdev=eth_af_packet0,iface=" + *device},
		LogType:  common.No,
	}
	flow.CheckFatal(flow.SystemInit(&config))

	// inputFlow, err := flow.SetReceiver(0)
	inputFlow, err := flow.SetReceiverOS(*device)
	flow.CheckFatal(err)

	flow.CheckFatal(flow.SetHandler(inputFlow, handler, nil))
	// flow.CheckFatal(flow.SetSender(inputFlow, 0))
	flow.CheckFatal(flow.SetSenderOS(inputFlow, *device))

	fmt.Printf("dev: %s, src: %s, dst: %s\n", *device, *src, *dst)

	flow.CheckFatal(flow.SystemStart())
}

func handler(current *packet.Packet, context flow.UserContext) {
	fmt.Printf("encapsulated packet %v", current)
	return
}

Then instead of using nff-go's methods to use AF_PACKET, I tried pass dpdk args to the framework (see the comment out lines above), and now the packets are processed, but both the original packet and the processed packet are sent through the interface.

07:18:19.443547 IP (tos 0x0, ttl 64, id 37111, offset 0, flags [DF], proto ICMP (1), length 84)
    50.50.50.50 > 40.40.40.40: ICMP echo request, id 1269, seq 1, length 64
07:18:19.443585 IP (tos 0x0, ttl 64, id 21506, offset 0, flags [none], proto ICMP (1), length 84)
    40.40.40.40 > 50.50.50.50: ICMP echo reply, id 1269, seq 1, length 64
07:18:19.443771 IP (tos 0x0, ttl 64, id 37111, offset 0, flags [DF], proto ICMP (1), length 84)
    50.50.50.50 > 40.40.40.40: ICMP echo request, id 1269, seq 1, length 64
07:18:19.443807 IP (tos 0x0, ttl 64, id 21506, offset 0, flags [none], proto ICMP (1), length 84)
    40.40.40.40 > 50.50.50.50: ICMP echo reply, id 1269, seq 1, length 64

You can see that both the icmp req and response are seen twice in the tcpdump above. The 2nd icmp req and response are the packets sent by this dpdk program and the first ones are the original packets.

I wonder why the original packet is also sent through the interface? Also it will be nice if I can get helps on figuring out the cause why I cannot use SetReceiverOS and SetSenderOS against a veth interface.

[gshimansky]

OS and DPDK functions are supposed to work independently and are designated for different network device types. If you bound your NIC to DPDK driver, use SetSender and SetReceiver, if you use Linux driver, then use SetSenderOS and SetReceiverOS. Conflating them will most likely cause problems and I am really surprised that you managed to make them both work :)