Is it possible to use AWS Step Functions to poll a DynamoDB Stream to reduce Lambda complexity?

Question

I am planning to use a DynamoDB Stream to do a bunch of task on a very complex single table. The task include actions such as sending notifications, updating other entries in the database and indexing some entries using AWS OpenSearch. I can already see that the Lambda function to handle all of this will end up being massively complex and very hard to maintain. It also feels very wasteful since most actions that would be included in this lambda only apply to very specific stream events and are not required for most others.

My thought was that I could maybe use a StateMachine to split these task up into individual Lambdas and then only run them when they are needed for a certain item in the stream.

So I'm curious if there is any easy way to poll a DynamoDB Stream with a Step Function in the same way you would do with a Lambda.

(PS. I know this could be considered an opinion question, but I would also be curious about anyone's thoughts on this solution. Is it a good idea or bad idea ?)

The word you are looking for is "poll" not "pole". Also, DynamoDB streams are a push mechanism, not a polling mechanism. — Mark B
– Mark B, Commented Feb 14, 2022 at 15:05
I don't see any way to send DynamoDB streams straight to a Step Function. It looks like you would need to send it to a Lambda function first, then if you don't want to do all the processing in the Lambda function you could push the data to a Step Function invocation. — Mark B
– Mark B, Commented Feb 14, 2022 at 15:05
The documentation clearly states AWS Lambda polls the stream and invokes your Lambda function synchronously when it detects new stream records. This makes me think DynamoDB streams are not a "push mechanism" but get polled externally. docs.aws.amazon.com/amazondynamodb/latest/developerguide/… — Mercury
– Mercury, Commented Feb 14, 2022 at 15:25

Hatim · Accepted Answer · 2022-02-14 15:14:12Z

1

The lambda must be unique for specific treatment, this is why the best practice is to split your lambda to many sub components

You can use this architecture for example :

DynamoDB Stream --> Lambda --> Step Function (parralele treatment)

Or use multiple lambdas with SNS fanout like :

DynamoDB Stream --> Lambda --> SNS --> multiple sub ( lambdas)

answered Feb 14, 2022 at 15:14

Hatim

1,1721 gold badge9 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Is it possible to use AWS Step Functions to poll a DynamoDB Stream to reduce Lambda complexity?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related