Gateway routing with multiple (public / private) network interfaces

[dedene]

I have the following setup: a cluster of mulitple servers at OVH where I want to run Talos. Each have access to a private network (VLAN based), with DHCP and a gateway for internet access. This works perfectly fine. There is a second network interface that can route public ips to the servers. You get a range and a associated gateway.

For the example, let's use following ips:

• 1.2.3.4 → my ip address
• 10.0.0.10/24 → the ip in the private network, configured through dhcp, on eth0.101 (so using vlan tag 101)
• 5.6.7.8 → a public ip in the /28 subnet leased from OVH, on eth0 (so same parent interface, untagged). The instructions from OVH are to use the second to last ip as Network Gateway: https://help.ovhcloud.com/csm/en-ca-dedicated-servers-ip-block-vrack?id=kb_article_view&sysparm_article=KB0043335 → Let's use 5.6.7.14/32 as the example.

This setup works great on a non-Talos Linus (Ubuntu or Debian), so it should be able to work on Talos as well.

When configuring a node as follows:

[...]
    networkInterfaces:
      - interface: eth0
        mtu: 1500
        dhcp: false
        addresses:
          - 5.6.7.8/32
        routes:
          - network: 5.6.7.14/32
          - network: 5.6.7.8/32
            gateway: 5.6.7.14
        vlans:
          - vlanId: 101
            dhcp: true
            routes:
              - network: 0.0.0.0/0
                gateway: 10.0.0.1
[...]

eth0 has 00:93:65:69:3d:bb as MAC address in this example.

When looking with tcpdump I can see package are reaching the Talos server fine, but the response seems to have the VLAN tag applied, so the reply is not routed correctly back.

12:03:04.887509 28:99:3a:99:9f:9e > 00:93:65:69:3d:bb, ethertype IPv4 (0x0800), length 98: 1.2.3.4 > 5.6.7.8: ICMP echo request, id 61214, seq 8, length 64
12:03:04.887580 00:93:65:69:3d:bb > 00:00:5e:00:01:02, ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4 (0x0800), 5.6.7.8 > 1.2.3.4: ICMP echo reply, id 61214, seq 8, length 64

Even when using metric: 0 for the public ip gateway route, it still has the VLAN tag applied.

Does someone has an idea if there is something missing from the config? Or what I can do to get the 5.6.7.8 ip address to respond?

[smira]

Your setup works exactly as you configured default gateway is 10.0.0.1 only. Why would packet to 1.2.3.4 be routed to 5.6.7.14 via eth0 untagged?

You need to figure out what should be the path for public outbound, if it's eth0, it should be a default route on eth0, and eth0.101 should only route to 10/8 ?

[smira]

I think it makes sense to create an issue to ask to support routing tables in routes and routing rules. We use them in Talos internally, but we have no external interface to configure them.

[dedene]

Ok, many thanks!! Shall I reference this discussion as an example use case?

For now, I'll try this: get an extension service to add the routing rule added when booting the node. I'll let you know if this works with MetalLB (as this probably is a also useful for others who want to use Talos on Bare Metal without external load balancer dependencies)

[dedene]

@smira I worked out a small extension service (zenjoy/talos-extensions@dc68d43) and it turns out to work fine with MetalLB (and allows me to make use of the OVH Vrack capabilities directly from Kubernetes).

Do you want me to make a PR to the extensions repo so others can make use of it? Or will you be able to add routing rules and tables to the Talos configuration in nearby future?

Meanwhile I added an issue for this feature request in #7184

[smira]

That looks really cool!

I would rather see if we can do an easier way, but to do that we first need to implement routing rules.

I don't know if OVH has a metadata service we could use to fetch these networking settings, and then Talos could use that to configure networking automatically. That's what we were doing with all other "clouds". So I think we should do routing rules first, once it's there, let's resync and see how we want to move forward with this.

[dedene]

Well the OVH public cloud certainly has a metadata service.

But their VRack thing is something else, very depending on how you want to use it. It allows for a virtual cross datacenter L2 network (with QinQ or Jumbo Frames etc, works very nice) for private networks and you can route public ipblocks towards it. So I don't think it's possible to automate that as it is very project specific.

I agree with what you say, I thinks it's indeed better to have it in the Talos config. I'm looking forward to test out when routing rules will be added natively to Talos. For now I think we can move forward with moving our last legacy projects (but that used this VRack feature) to Talos/Kubernetes.

Thanks for the help, enjoy the weekend!

[steverfrancis]

Yes, I think it would help if you posted your actual machine config, showing the route for the 1.2.3.4 address/network.

[dedene]

Also many thanks for your help! I posted the config here. I may have overlooked the routing table parts at first, as ping did work without these post-up commands. We're coming from a legacy config using keepalived, so that's how I missed that.

I'm happy to share the real config and ips on a more private channel (i.e. Slack DMs?) if that helps?