Putting already encoded DynamoJSON leads to double base64-encoded binary values

[thefirstofthe300]

Describe the bug

I'm working on migrating some data in Dynamo into a new Dynamo account. I need to create a synchronization mechanism to prevent keep data up to date while I work on getting things cut over. The sync pipeline we've built involves using DynamoDB streams -> Lambda to update the new database.

The record fed to the lambda is a Python dict with the proper structure and encodings. None of the binary values are decoded from base64. I would expect that writing that record to the database using the boto3 client's put_item() function should send the data as is considering that the data isn't being decoded when being sent from Dynamo. However, in this case, it appears that binary values are base64 encoded by boto3 first.

Expected Behavior

boto3 shouldn't encode any values fed to the dynamodb client's put_item() function.

Current Behavior

boto3 is encoding binary values using base64 when using put_item()

Reproduction Steps

Run the code in this repo: https://github.com/thefirstofthe300/boto3-binary-encoding

The record.json contains a properly encoded record as the Lambda is receiving from DynamoDB streams.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3==1.24.2; botocore==1.27.2

Environment details (OS name and version, etc.)

macOS 12.4 with python 3.9.13

[tim-finnigan]

Hi @thefirstofthe300 thanks for reaching out. It looks like this overlaps with an older issue in the CLI repository: aws/aws-cli#1097. Maybe you could try using one of the approaches in this comment.

[tim-finnigan]

We can continue tracking this in the issue mentioned above (aws/aws-cli#1097). Both the AWS CLI and boto3 are built on botocore (https://github.com/boto/botocore) so this relates to both repositories. I'll close this in favor of that issue since as it is older and has more comments.