Deserialize DDB item from JSON

[mbergkvist]

Can i use serde_dynamo to deserialize an item in JSON to a struct. Something like

{
  "prop1": {
    "S": "value1"
  },
  "prop2": {
    "S": "value2"
  }
}

to

struct MyStruct {
  prop1: String,
  prop2: String,
}

The reason I'm asking is that I'm trying to deserialize the newItem and oldItem in a DDB Stream event.

[mbergkvist]

My current working plan is to add support in serde_dynamo for the aws-lambda-events AttributeValue. Does that make sense?

[bryanburgers]

Since StreamRecord.old_image and .new_image are already converted to HashMap<String, AttributeValue>, then yes, I would think supporting AttributeValue would be enough.

You shouldn't have to handle the AWS's special serialization of DynamoDb values

{
  "prop1": {
    "S": "value1"
  },
  "prop2": {
    "S": "value2"
  }
}

It looks like aws_lambda_events is already handling that part.

[mbergkvist]

As a first attempt, I added support for aws_sdk_dynamodbstreams::model::AttributeValue, which was trivial, reusing what was already done for aws_sdk_dynamodb::model::AttributeValue and that the types are identical.

However, implementing the AttributeValue trait for aws_lambda_events::dynamodb::attributes::AttributeValue was not that simple (probably due to my lacking Rust skills). I'm stuck on fn as_ns(&self) -> Option<&[String]> and fn as_n(&self) -> Option<&str>, where aws_lambda_events::dynamodb::attributes::AttributeValue::Number contains a f64 but I can't figure out how to return &str. Any suggestion? The rest of the serde_dynamo::attribute_value::AttributeValue is implemented.

[mbergkvist]

The implementation is here, mbergkvist#1
As mentioned previously, as_n() and as_ns() is not implemented, and there is an unwrap() in construct_n() I would like to remove. However, I can't figure out how to solve any it. Any suggestions @bryanburgers? Any help is appreciated.

[bryanburgers]

Related: awslabs/aws-sdk-rust#497

[indiv0]

Any suggestions for how we can get mbergkvist#1 over the finish line? This PR would be super useful.

[bryanburgers]

version 4.0.0 was released this morning with support for aws_lambda_events.

4.0.0 doesn't implement deserializing from an item in JSON format, but does the 4.0.0 release fulfill your needs?

[indiv0]

@bryanburgers yes it looks like 4.0.0 supports my needs, thank you very much! Could you elaborate on that JSON deserialization being missing though?

[bryanburgers]

@bryanburgers yes it looks like 4.0.0 supports my needs, thank you very much! Could you elaborate on that JSON deserialization being missing though?

@indiv0, yeah, by that I just mean that serde_dynamo doesn't deserialize from something like

{
  "prop1": {
    "S": "value1"
  },
  "prop2": {
    "S": "value2"
  }
}

into an AttributeValue. That's not a change in 4.0.0; it has never done that part of the deserialization. That part of the deserialization has always been handled by aws-sdk-dynamodb, rusoto_dynamodb, and aws_lambda_events.

[mbergkvist]

Thanks for the support!

[mlafeldt]

Seriously though, it shouldn't be so hard to map JSON to Vec<HashMap<String, AttributeValue>> and vice versa.

I tracked down the generated code in the AWS SDK that does it:

https://github.com/awslabs/aws-sdk-rust/blob/5848182a81ac2e63e3e54629c56d01eec047e098/sdk/dynamodb/src/json_ser.rs#L1900

However, that code isn't exported, so no way to use it. :/

(Would be awesome if the SDK's AttributeValue would just implement Serialize of course.)

Anyway, I really like serde_dynamo. Would you be open to adding support (accepting a PR) for JSON? Could come in the form of a simple helper function similar to the linked code. I know it's not the primary goal of this project. Let me know what you think.

[mlafeldt]

(Would be awesome if the SDK's AttributeValue would just implement Serialize of course.)

That's exactly what Rusoto does btw: https://rusoto.github.io/rusoto/src/rusoto_dynamodb/generated.rs.html#40-91

[bryanburgers]

@mlafeldt If this code was added to this project, how would you use it?

You've basically make two assertions here:

• It would be easy to add this
• It would be cool to add this

I agree with both. (That argument also applies to a lot of other things that haven't been added: an item! macro like serde_json's json! macro, deserializing using borrowed data instead of owned data, a variety of other little experiments I've run...)

But unless the code adds tangible value to the project, then adding this code is adding to the maintenance of the code without adding value, and that's a net loss.

Right now, my understanding is that the only way this library gets used is:

• When retrieving data from DynamoDb (at which point aws-sdk-dynamodb already does the deserializing)
• When a lambda is run with a DynamoDb event (at which point aws_lambda_events already does the deserializing)

Do you have a different use for this library?

I'm certainly not saying no. I've gone back and forth on adding this a couple of times. What I am saying is that I'd like to understand the real world use-cases first, and go from there.

[mlafeldt]

Hey @bryanburgers,

I agree with your points, esp. the one on maintainability.

I did a bit more digging and the situation with serde and aws-sdk-rust seems, well, complicated.

In the meantime, I was able to marshall/unmarshall DynamoDB items to and from JSON by piecing together code from Rusoto and serde_dynamo.

(My original use case involved using serde_dynamo to turn items from the SDK into serde_json::Value, which can be passed to other serde implementations for further encoding.)

[lamalex]

@bryanburgers I came here looking for the same thing, my use case is different, however. I'm using a sort key to serialize a bunch of information about the item stored in my table, I have a parser written in rust that deserializes that information into fields on a struct.

I am now trying to share that parser code with a node application that uses the same table and has its own much slower, much more fragile parser implementation.

The AWS dynamodb sdk isn't usable within WASM, so I'm using the node SDK to fetch my items as JSON. I want to pass the JSON across the WASM boundary to do the conversion into my domain object.

[bryanburgers]

Oooh, that’s a fantastic use case.

[lamalex]

I started writing a separate crate with an entire json parser/deserializer for the dynamodb attributevalue format, but perhaps you have a better idea?

[lamalex]

@bryanburgers can we reopen this issue, or would you like me to open a new one

[bryanburgers]

Anyone who's still interested in this issue: I just pushed a branch up, and I'd be curious if you could use that branch and see if it works for you?