The security token included in the request is expired

[victoramsantos]

Hi, I running prometheus-elasticache-sd with the last version in a Kubernetes cluster (v1.28.5-eks) which add ServiceAccount to the pod using kube2iam.
I started to see some error messages related to ExpiredToken: The security token included in the request is expired, like this one:

ts=2024-05-23T21:00:28.336Z caller=main.go:325 level=info msg="Starting prometheus-elasticache-sd" version="(version=1.0.291, branch=main, revision=e27b3eae045857eb3fe9db27376e9fac82c3d752)"
...
ts=2024-05-24T00:01:31.622Z caller=main.go:234 level=warn msg="could not list tags" err="operation error ElastiCache: ListTagsForResource, https response error StatusCode: 403, RequestID: reqid, api error ExpiredToken: The security token included in the request is expired" ARN=arn status=available

After some research I found that aws-sdk-go-v2 doesn't refresh the AWS token, it just loads and leaves it in memory (ref).

Would be possible to update prometheus-elasticache-sd to sporadically refresh this credential?

[maxbrunet]

Hey @victoramsantos, thank you for reporting this and the initial research, I would recommend using the first-party IAM roles for service accounts over kube2iam, and the issue would not occur.

Otherwise, are you seeing errors containing both could not describe cache clusters and ExpiredToken too?

If so, adding apiErr.ErrorCode() == "ExpiredToken" to this condition would drop the client when it occurs, and the next refresh would create a new one, effectively re-reading the credentials:

prometheus-elasticache-sd/main.go

Lines 205 to 207 in 6607b63

	if errors.As(err, &apiErr) && (apiErr.ErrorCode() == "AuthFailure" || apiErr.ErrorCode() == "UnauthorizedOperation") {
	d.elasticache = nil
	}

If you are willing to test that and open a PR for it, I'd be open to merge it :)