podman is overwriting user's CNI_ARGS to the CNI plugin

[space88man]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman seems to be overwriting the user supplied CNI_ARGS so some settings are not transmitted to the CNI plugin.

When dumping the environment of a plugin I see

CNI_ARGS='IgnoreUnknown=1;K8S_POD_NAMESPACE=abcd_3;K8S_POD_NAME=abcd_3;K8S_POD_INFRA_CONTAINER_ID=4c7f1903383a9552973ba2deb5ceb2e8b65ab1e06309c8af948b217724718bc3'

instead of the value provided.

When using a user network (pre existing bridge) , the CNI_ARGS environment variable doesn't assign a static IP address. This affects both host-local and static plugins.

Steps to reproduce the issue:

1. Create a user network: this is an existing (libvirt) bridge on the host, so omit masq gateway iptables settings

# /etc/cni/net.d/85-network-125.conflist
{
    "cniVersion": "0.4.0",
    "name": "network-125",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "vmbr0",
            "ipam": {
                "type": "host-local",
                "routes": [
                    {
                        "dst": "0.0.0.0/0"
                    }
                ],
                "ranges": [
                    [
                        {
                            "subnet": "192.168.125.0/24"
                        }
                    ]
                ]
            }
        }
    ]
}

2. Create container with static IP address

[root@amadeus ~]# CNI_ARGS=ip=192.168.125.123 podman run  --rm --network network-125  -it  --dns 192.168.125.1   centos:8
[root@5ddaa9f68d57 /]# ip -4 addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if66: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default  link-netnsid 0
    inet 192.168.125.17/24 brd 192.168.125.255 scope global eth0
       valid_lft forever preferred_lft forever

Describe the results you received:
The IP address is randomly assigned by host-local IPAM. CNI_ARGS is not respected by plugin.

Describe the results you expected:
Static IP address assignment from CNI_ARGS environment variable.

Additional information you deem important (e.g. issue happens only occasionally):
Try with and without network prefix; lowercase and all caps.

Output of podman version:

1.6.2-1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.2-2.fc31.x86_64
containernetworking-plugins-0.8.2-2.1.dev.git485be65.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

1. Fedora 31
2. The bridge is configured by and shared with libvirtd, so omitted most of the configuration of CNI

[space88man]

IPAM static is also not working with CNI_ARGS.

Configure a minimal static network

{
    "cniVersion": "0.4.0",
    "name": "static",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "vmbr0",
            "ipam": {
                "type": "static",
                "routes": [
                    {
                        "dst": "0.0.0.0/0"
                    }
                ],
                "addresses": [
                    {"address": "192.168.125.123/24", "gateway": "192.168.125.1"}
                ],
                "dns": {
                     "nameservers": ["192.168.125.1"]
                 }
            }
        }
    ]
}

Try changing the IP address: CNI_ARGS="IP=192.168.125.200;GATEWAY=192.168.125.1" podman run ... — this does not work. Only the IP 192.168.125.123 from the configuration file is used.

The addresses array is supposed to be optional. However if it is omitted and an IP address is attempted with CNI_ARGS

INFO[0000] About to add CNI network testing (type=bridge) 
ERRO[0000] Error adding network: IPAM plugin returned missing IP config 
ERRO[0000] Error while adding pod to CNI network "testing": IPAM plugin returned missing IP config 

[space88man]

The user's CNI_ARGS seems to be replaced by podman with

CNI_ARGS='IgnoreUnknown=1;K8S_POD_NAMESPACE=abcd_3;K8S_POD_NAME=bright_curie;K8S_POD_INFRA_CONTAINER_ID=342853d0e227a2f5cbc0bcfd6642983217c3b848e31ab2d2a0245f2ef2edcaae'

[space88man]

The following hack restores the "correct" behaviour:

mv  /usr/libexec/cni/static  /usr/libexec/cni/static.real
cat > /usr/libexec/cni/static  <<EOF
#! /bin/bash
CNI_ARGS="$CNI_ARGS;$CNI_EXTRA_ARGS"
exec /usr/libexec/cni/static.real
EOF
chmod +x /usr/libexec/cni/static

Then

# this actually works as it adds the users' CNI_ARGS back to the
# podman overwritten ones!
CNI_EXTRA_ARGS="IP=192.168.125.200/24;GATEWAY=192.168.125.1" \
    podman run --network static ...

This works (!).

[rhatdan]

@baude @mheon Can we extend the ARGS rather then truncating them?
@space88man Any change you would want to open a PR?

[mheon]

This is all in OCICNI - we don't touch any of it in Libpod land

[rhatdan]

@dcbw FYI

[space88man]

@rhatdan happy to PR - where is the "upstream" is it pkg/ocicni here or the separate project https://github.com/cri-o/ocicni. (Ref: cri-o/ocicni#37 (comment) "Consider moving this project into CRI-O")

[mheon]

@space88man The upstream is github.com/cri-o/ocicni

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

I think this is still an issue.

[vrothberg]

Still an issue.

[rhatdan]

Is this our issue or CNI?

[vrothberg]

CNI -> cri-o/ocicni#65

[space88man]

@rhatdan cri-o/ocicni#70 PTAL, does this work for podman?
Since the real issue is with ocicni, I guess you should close this issue. Thanks.

[rhatdan]

I will let @mheon and @baude answer that question.

[baude]

we w ould just need to know to vendor in when it is merged

[rhatdan]

This Issue is like the never ending story...