external-dns crashlooping when using godaddy provider - panic: invalid argument to Int63n #4864
Comments
|
I am currently experiencing the same problem. As a result of testing with the same API key using postman, I received the following response.
{
"code": "QUOTA_EXCEEDED",
"message": "The monthly quota for requests has been exceeded."
}I was also able to confirm that when the monthly quota limit is hit, the retryAfter header is not included. FYI: Since I have a one-month limit, it is currently impossible to check whether the retryAfter header comes in case of a one-minute limit. This appears to result in the retryAfter header being pulled as blank and parsed as 0, causing a panic in the I didn't know whether the one-month limit existed originally, but it seems that the problem of not having a retryAfter header can be solved as follows. // provider/godaddy/client.go:224
func (c *Client) Do(req *http.Request) (*http.Response, error) {
...
resp, err := c.Client.Do(req)
// In case of several clients behind NAT we still can hit rate limit
for i := 1; i < 3 && err == nil && resp.StatusCode == 429; i++ {
retryAfter, _ := strconv.ParseInt(resp.Header.Get("Retry-After"), 10, 0)
var jitter int64
if retryAfter > 0 {
jitter = rand.Int63n(retryAfter)
}
retryAfterSec := retryAfter + jitter/2
...
}In the case of a one-month limit, it seems that the body message must be read to distinguish correctly, but since there appears to be a function that already reads the body message after the // provider/godaddy/client.go:224
func (c *Client) Do(req *http.Request) (*http.Response, error) {
...
resp, err := c.Client.Do(req)
// In case of several clients behind NAT we still can hit rate limit
for i := 1; i < 3 && err == nil && resp.StatusCode == 429; i++ {
var retryAfter int64
var jitter int64
if headerValue := resp.Header.Get("Retry-After"); headerValue == "" {
// Even with the 1 minute limit, there may be no retryAfter header, so set it to 30 seconds instead of 60 seconds.
retryAfter = 30
} else {
retryAfter, _ = strconv.ParseInt(headerValue, 10, 0)
if retryAfter > 0 {
jitter = rand.Int63n(retryAfter)
}
}
retryAfterSec := retryAfter + jitter/2
...
}I will pull request this code. Please let me know if anything needs to be fixed |
|
@alexstojda Of course. I will discuss this in the PR |
What happened:
Start around 11:55 EST today (Monday, Nov 11th, 2024) my external DNS deployment started crashlooping out of the blue. No configuration change has been made in the last 2 weeks.
As far as I can tell, it seems to be failing because the
Retry-Afterheader is either missing or its value can't be parsed to Int64 here:external-dns/provider/godaddy/client.go
Line 233 in ea6d44b
In addition, what is strange is that the 429 response, which is what would trigger the above code, only seems to happen in the external-dns controller. Calling the GoDaddy API with the same API Key/Secret from the same IP does not trigger a 429 response, yet the external-dns pod is consistently unable to start.
What you expected to happen:
External-dns should not crash. At best, there should be a more graceful handling of a missing or invalid
Retry-Afterheader. Maybe defaulting to 60 seconds?How to reproduce it (as minimally and precisely as possible):
Still unclear what the root cause is. Will setup a local cluster and report back if I find anything.
Anything else we need to know?:
Logs:
Environment:
external-dns --version): 1.15.0 (also happening in 1.14.5)The text was updated successfully, but these errors were encountered: