Nginx resolver and IPv6 in container (podman)

[gitouche-sur-osm]

Hello

I know podman is not supported, but we have been running jitsi in podman flawlessly for a long time.
The fix to nginx resolver was very much needed for podman to continue working (thank you!).

After upgrading to 8044, I was surprised to find jitsi-meet still could not resolve the prosody server name in nginx internal resolver.
I found a new error in Nginx logs : unexpected A record in DNS response and figured it was expecting another type of DNS record (AAAA?).

The containers in podman now have full IPv6 support (even when not using it, the containers have a fe80:: Link-local IPv6 address) and it changes nginx behaviour while resolving DNS names.

We resolved this by adding ipv6=off to the resolver configuration in

docker-jitsi-meet/web/rootfs/defaults/nginx.conf

Line 30 in c44c59e

resolver {{ .Env.NGINX_RESOLVER }};

This is the first time we had to alter the image building process to make docker-jitsi-meet work with podman.
Would you consider adding ipv6=off to your nginx.conf template file? I don't think it will break any docker configuration.

Thank you

[saghul]

An env variable that allows to disable it would be acceptable, while keeping the current behavior by default.

[gitouche-sur-osm]

An env variable that allows to disable it would be acceptable, while keeping the current behavior by default.

That would be perfect. I did some more testing and found no other way to fix the issue.
I would be glad to try to provide a PR for an env variable. What name should it use?

[saghul]

How about DISABLE_IPV6 ?

[benz0li]

How about DISABLE_IPV6 ?

Please don't. There is already ENABLE_IPV6, which is used for web and prosody.

I suggest renaming ENABLE_IPV6 to ENABLE_LISTEN_IPV6 (default: "true") and naming the new env variable DISABLE_RESOLV_IPV6 or DISABLE_RESOLVE_IPV6 (default: "false").

P.S.: I work around this issue by setting NGINX_RESOLVER=127.0.0.11 ipv6=off.

[saghul]

Ah I had forgotten about that one, thanks for pointing it out!

Btw, If there is already a way to work around it, not sure we need a new one...

[gitouche-sur-osm]

I can't use that, as the container resolver IP address cannot be predicted and is copied over from the local /etc/resolv.conf to nginx configuration (see 53b2654).

[saghul]

Right. Then yeah we need an extra env variable.

Maybe we can be exploit and call it NGINX_RESOLVER_DISABLE_IPV6

[benz0li]

Maybe we can be exploit and call it NGINX_RESOLVER_DISABLE_IPV6

NGINX_RESOLVER_DISABLE_IPV6 is perfectly fine, too.

[gitouche-sur-osm]

I finally found the root cause for nginx behaviour.
Following this comment, I found a bug in the DNS resolver podman uses "aardvark-dns" : it answers A records when asked for AAAA records and this is not expected behaviour. Nginx errors out on it.

This got fixed in aardvark-dns 1.2.0 as coredns: do not combine results of A and AAAA records. I am using aardvark-dns 1.1.0 at the moment!

In my case, the needed conditions for the bug to appear are :

• podman 4.x (current) with netavark+aardvark-dns network stack (this is the new network stack, default is still CNI)
• RHEL/CentOS 8 and clones. Fedora and RHEL/CentOS 9 already have a fixed aardvark-dns (1.3.0)!

Hopefully RHEL/CentOS 8 will get the news aardvark-dns 1.3.0 in future release 8.8 (6 months from now).

Until then, I can live with patching ipv6=off on top of the repo to build the images. I'm not sure anyone else will ever encounter the bug, so feel free to close the issue if you think it better. I would use a env. variable if it was there but the root cause is not with docker-jitsi-meet 👍

[saghul]

Good analysis, thanks!