long running s3 cp should automatically fetch renewed credential

[mdavis-xyz]

Steps to reproduce

1. Have a bucket with a lot of files.
2. Have an environment where credentials get rotated occasionally. e.g. Cloud9, where there's always a valid credential in ~/.aws/credentials, but it gets changed every 15 minutes or so
3. Do a long running copy, that takes more than 15 minutes. aws s3 cp s3://bucket/prefix/ s3://otherbucket/prefix --recursive

expected behavior

Since there are always valid credentials in ~/.aws/credentials, the cli copies all the files over without error.
Specifically, when it encounteres an auth error because the credentials it grabbed at the beginning of the operation have expired, check the credential file again, note that the contents have changed (e.g. file modification timestamp, or different contents, or load the contents and do an API call), then continue copying the remainder of files with the new credential.

actual behavior

move failed: s3://b/prefix/2021-03-28-18-25-36-9AE1F806D0BAC06E to s3://b2/prefix/2021-03-28-18-25-36-9AE1F806D0BAC06E An error occurred (ExpiredToken) when calling the CopyObject operation: The provided token has expired.
fatal error: An error occurred (ExpiredToken) when calling the ListObjectsV2 operation: The provided token has expired.

The operation sucessfully copied/moved files for 15 minutes or so, then the existing credentials expired, and the cli aborted the task. Many files remain unmoved/uncopied. Even though the credentials in ~/.aws/credentials at the time of failure were valid.

For a copy in particular, there's no easy way to pick up where you left off. You have to re-copy the same files, and then it will fail at the same point again.

[kdaily]

Hi @mdavis-xyz,

Thanks for your post. What you're experiencing is a duplicate of this issue: #9034

We only read credentials from the credential file once. There are a number of other refreshable credential mechanisms that are more robust than putting credentials in the credential file. I understand if you're using a system like Cloud9 that rewrites credentials to the shared credential file. Changing this behavior requires discussion across all AWS SDKs, hence why it is in that other repository. I don't currently have a timeline on when any decisions about if or when supporting that use case would be made.

Since this is known behavior tracked elsewhere, I'm going to close this issue. You can upvote or comment on the linked issue to add to the conversation!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[kdaily]

Hi @mdavis-xyz,

One update: Cloud9 does support using other temporary credential methods in certain environments on EC2s, as described here:

https://docs.aws.amazon.com/cloud9/latest/user-guide/credentials.html

I'm not seeing in their documentation that they support or recommend rewriting the shared credential file anymore. If you have a reference for where that still exists, I'd appreciate a link!