Is there a way to convert numbers to the strings that DynamoDB expects in Step Functions?

Question

I have an IoT Topic receiving data from devices. Each IoT payload includes some properties and an array of objects, which looks something like this.

{
  "batchId": "someBatchId",
  "prop1": "someProp1",
  "objArray": [
    {
      "arrString1": "someArrString1",
      "arrString2": "someArrString2",
      "arrNum1": 1,
      "arrNum2": 2,
      "arrString3": "someArrString3"
    },
    {
      "arrString1": "someArrString4",
      "arrString2": "someArrString5",
      "arrNum1": 3,
      "arrNum2": 4,
      "arrString3": "someArrString6"
    }
  ]
}

The array can have hundreds of objects in it. We want to flatten this data out using a Map step and associate the top-level properties with each element in the array and insert that element into DynamoDB. We have the table set-up and the IoT topic working just fine.

The problem we have is that DynamoDB expects strings when inserting numbers. However, since we're receiving this data as a JSON object from IoT and the numbers are inside of the array of objects, we're having a hard time massaging the numbers into strings. So, we want the Step Function to convert the numbers into strings somehow, but I can't see how to do it. The goal here is to build a simple pipeline for storing IoT data into DynamoDB.

We also don't fully control all of the properties that could be sent, so we're also storing copies of the IoT payloads in S3 (which is already wired with the IoT rules engine and works just fine), but this is more of a backup and catch-all. We're mostly interested in the data getting into DynamoDB so that we can actually query it. How can we convince the Step Function to insert the numbers from the JSON payloads into DynamoDB?

Have you considered using the IoT rule SQL to convert the numbers to strings? A nested SELECT VALUE query might be able to do this. — cementblocks
– cementblocks, Commented Aug 13, 2020 at 1:00
@cementblocks I have. I'm not a fan because if our clients end up sending additional data, I don't want to exclude that data because I had to select individual properties instead of *. This is essentially a simple pipeline to ingest data for analysis so that we can derive a solution for what the data represents. — Azhdeen
– Azhdeen, Commented Aug 13, 2020 at 6:08
I would just pivot to a lambda either in the step function or as the target of the topic rule. — cementblocks
– cementblocks, Commented Aug 13, 2020 at 17:26
@cementblocks: Yeah. Small rant: AWS makes some interesting tools, but seems to overlook small details that breaks integrations, and the expectation is always "Well, just throw code at it in a Lambda" and it's annoying. But having said that, thinking about your sub-query suggestion more, it could work as long as having some of the properties duplicated in the objects is OK. That way, you can still SELECT * on the objects to catch any extra data. Might get complicated with character quotas though depending on the number of objects in the array. — Azhdeen
– Azhdeen, Commented Aug 13, 2020 at 18:18
This just hit: aws.amazon.com/blogs/aws/… Maybe use the string formatting to add quotes around the number? — cementblocks
– cementblocks, Commented Aug 14, 2020 at 2:05

NickC · Accepted Answer · 2021-12-06 22:56:31Z

You are really asking two questions here.

Can Amazon States Language convert numbers to strings?
How can you get Step Functions to add things to Dynamo DB without specifying the data type.

The answer to the first question is that yes, you can use ASL to convert numbers to strings using the concatenation intrinsic function like so. Given a payload of a single number

{
  "key.$": "States.Format('{}',$.Payload)"
}

We can use the States intrinsic function Format to add quotes around the output of this step.

This will not be helpful in your use case however, as you have hundreds of numbers potentially which may not always follow a set format.

In your case, the solution would be to save your data to DynamoDB using a Lambda function with the Document Client.

It would be nice if they had an option to use the document client directly within Step Functions, but as of this writing that is not the case. You simply need to perform the action manually within a Lambda function using the document client. Same result.

Collectives™ on Stack Overflow

Is there a way to convert numbers to the strings that DynamoDB expects in Step Functions?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related