S3 PutObjectAsync() throws on duplicate trailing header key when SDK retries the request

[dorny]

Describe the bug

S3 PutObjectAsync() throws:

System.ArgumentException: An item with the same key has already been added. Key: x-amz-checksum-crc32c

This consistently happens when:

• ChecksumAlgorithm is set to ChecksumAlgorithm.CRC32C or any other additional checksum algorithm
• Checksum is calculated by SDK and send via trailing headers, no pre-calculated value is provided
• SDK retries the request (e.g. first request fails due to timeout or some other transient error)

Expected Behavior

PutObjectAsync() with checksum algorithm (e.g. CRC32C) can be retried by SDK without throwing exception.

Current Behavior

IAmazonS3.PutObjectAsync() with ChecksumAlgorithm set to CRC32C throws System.ArgumentException when the request is retried by SDK:

Stack trace:

System.ArgumentException: An item with the same key has already been added. Key: x-amz-checksum-crc32c
   at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
   at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
   at Amazon.Runtime.Internal.Util.ChecksumUtils.SetRequestChecksum(IRequest request, String checksumAlgorithm, Boolean fallbackToMD5)
   at Amazon.Runtime.Internal.ChecksumHandler.PreInvoke(IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ChecksumHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CredentialsRetriever.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.RetryHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.CallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.S3.Internal.AmazonS3ExceptionHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.ErrorCallbackHandler.InvokeAsync[T](IExecutionContext executionContext)
   at Amazon.Runtime.Internal.MetricsHandler.InvokeAsync[T](IExecutionContext executionContext)

Reproduction Steps

using Amazon.Runtime;
using Amazon.S3;
using Amazon.S3.Model;

// Credentials taken from ENV  
var s3 = new AmazonS3Client(new AmazonS3Config
{
    // Unrealistic timeout so SDK will do a retry
    Timeout = TimeSpan.FromMilliseconds(1),
    RetryMode = RequestRetryMode.Standard,
    MaxErrorRetry = 1
});


// Replace the bucket name with some existing writable bucket
const string bucketName = "SOME_BUCKET"; 
const string objectKey = "s3-retry-bug";

// Random data; 
var data = new byte[1024];
Random.Shared.NextBytes(data);

try
{
    await s3.PutObjectAsync(new PutObjectRequest
    {
        BucketName = bucketName,
        Key = objectKey,
        ChecksumAlgorithm = ChecksumAlgorithm.CRC32C,
        InputStream = new MemoryStream(data)
    });
}
catch (ArgumentException)
{
    Console.WriteLine("Bug reproduced");
}

Possible Solution

I think the exception is thrown from here:

aws-sdk-net/sdk/src/Core/Amazon.Runtime/Internal/Util/ChecksumUtils.cs

Line 83 in a55fc1e

request.TrailingHeaders.Add(checksumHeaderKey, string.Empty);

It could probably work if it either checks if the key already exists, or use a setter instead of Add().

Additional Information/Context

No response

AWS .NET SDK and/or Package version used

AWSSDK.S3 3.7.305.7

Targeted .NET Platform

.NET 7

Operating System and version

mcr.microsoft.com/dotnet/aspnet:7.0 running in AWS EKS

[dscpinheiro]

Thanks for reporting this problem, we merged a fix earlier today and it's included in the last version of the S3 package (3.7.305.19 as of today).

[github-actions]

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.