Haproxy ingress doesn't report correct error messages or have a retry when an reload issue happens

[Dan-Jia-SAP]

Description of the problem

We noticed that when haproxy reload faces an issue, especially when multiple reloads happen at the same time
-> haproxy ingress logs show that the communication to the master is refused:
error reloading server: error sending reload to master socket: error connecting to /var/run/haproxy/master.sock: dial unix /var/run/haproxy/master.sock: connect: connection refused
but also:
10 controller.go:366] finish haproxy reload id=2: reload_haproxy=5000.375786ms total=5000.375786ms,
which is very confusing, sounds like the haproxy reload finished successfully in 5 sec.
But actually, the reload was not successful and there was no retry.
-> In this case, the failed/latest haproxy config update cannot be triggered successfully, they are not able to reach out to the backends, until the next reload succeeds again.

Expected behavior

• Once a newer haproxy reload is refused because the last haproxy reload has not finished, the failed reload should be reported and a retry should be triggered between the lines. There could be some messages like haproxy reload failed, will retry again
• Making the timeout setting for sending the socket configurable

Steps to reproduce the problem

In our cases, the reload issues happen normally when a second reload is triggered when another reload takes longer to get a response and has not finished...
Therefore, we reproduce the issue by:

1. Delaying the startup of each new haproxy process (for example, by adding a sleep statement here
2. Tigger the first reload process, and before it finishes, trigger the second haproxy reload
3. haproxy reload logs then show error reloading server: error sending reload to master socket: error connecting to /var/run/haproxy/master.sock: dial unix /var/run/haproxy/master.sock: connect: connection refused, and the haproxy connections cannot be routed to the backends, until a following reload succeeds again

Environment information

HAProxy Ingress version: v0.14 and HAProxy version we are using: v2.9.0

Command-line options:

(paste command-line options here)

Global options:

(paste global configmap options here)

Ingress objects:

(paste ingress objects here)

[jcmoraisjr]

Hi, can you confirm if this happens with a haproxy version older than v2.9, some v2.8 maybe? It seems we have some compatibility issue with newer versions which is missing to be addressed.