Unauthorized during apply when using a plan-file and AWS kubernetes token.

[perj]

Terraform Version, Provider Version and Kubernetes Version

Terraform version: 1.3.6
Kubernetes provider version: 2.19.0
Kubernetes version: v1.22.17

Affected Resource(s)

It's a general provide issue.

Terraform Configuration Files

provider "aws" {
  region = "eu-north-1"
  assume_role {
    role_arn = "arn:aws:..."
  }
}

locals {
  cluster_name = "mycluster"
}

data "aws_eks_cluster" "cluster" {
  name = local.cluster_name
}

data "aws_eks_cluster_auth" "cluster" {
  name = local.cluster_name
}

provider "kubernetes" {
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
  token                  = data.aws_eks_cluster_auth.cluster.token
}

resource "kubernetes_cluster_role" "test" {
  metadata {
    name        = "test"
  }
}

Debug Output

https://gist.github.com/perj/4783ccfa69812f1eb0fb329bb8fc9aa0

Panic Output

Steps to Reproduce

1. terraform plan -out=plan
2. sleep $((20 * 60))
3. terraform apply plan

Expected Behavior

Successful apply

Actual Behavior

Error from apply:

Error: Unauthorized

Important Factoids

We're using Atlantis. The plan is done when the PR is opened, the apply after it has been approved, which can take several hours.

References

This is basically a request to re-open #918
We can't use the workaround suggested there because we do an assume-role in the AWS provider so the same credentials are not available outside of the Terraform process.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[perj]

I'm using this script as a workaround right now, called via exec in the Kubernetes provider.

#!/usr/bin/env bash

# This script repeats the step done by the AWS provider, to arrive at
# the right AWS account and then print a token.
# This token is shortlived, but a new one is created both during plan
# and during apply, so it shouldn't matter.

ACCOUNT_ID="$1"
CLUSTER_NAME="$2"
ROLE_NAME=myrole
SESSION_NAME=AtlantisEKSToken

eval $(aws sts assume-role --role-arn "arn:aws:iam::$ACCOUNT_ID:role/$ROLE_NAME" --role-session-name "$SESSION_NAME" | jq -j '.Credentials | "token_AWS_ACCESS_KEY_ID="+.AccessKeyId+";token_AWS_SECRET_ACCESS_KEY="+.SecretAccessKey+";token_AWS_SESSION_TOKEN="+.SessionToken')
test -n "$token_AWS_SESSION_TOKEN" || exit 1

export AWS_ACCESS_KEY_ID="$token_AWS_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$token_AWS_SECRET_ACCESS_KEY"
export AWS_SESSION_TOKEN="$token_AWS_SESSION_TOKEN"

aws eks get-token --cluster-id "$CLUSTER_NAME" --output json

[jeremymcgee73]

Bumping, I'm having the same problem, even when using exec.

[emcay]

Same problem, using any configuration method.

[emcay]

For anyone using roles, and having persistent issues check this out. This ended up being the cause of my problems.

kubernetes-sigs/aws-iam-authenticator#268

[florianmutter]

We seem to have the same issue but with google authentication to kubernetes as described here: https://registry.terraform.io/providers/hashicorp/google/latest/docs/guides/using_gke_with_terraform#using-the-kubernetes-and-helm-providers

[manu-sucerquia-payu-gpo]

Have the same issue using exec and the roles seem to be ok. Any idea on any other permanent workaroud?