Dynamodb streams in python

Question

I would like to read data from a dynamodb stream in python and the alternatives that i have found so far are

Use dynamodb stream low level library functions (as described here): This solution however seems almost impossible to maintain in a production environment, with the application having to maintain the status of shards, etc.
Use KCL library designed for reading Kinesis streams: The python version of the library seems unable to read from a dynamodb stream.

What are the options to successfully process dynamodb streams in python? (links to possible examples would be super helpful)

PS: I have considered using lambda function to process the dynamodb but for this task, I would like to read the stream in an application as it has to interact with other components which cannot be done via a lamda function.

Curious as to which solution you went with? I'm currently facing the same dilemma and am considering implementing my logic in Java using KCL. — Matt Fortier
– Matt Fortier, Commented Jun 20, 2016 at 8:29
Any updates on this? I am considering going the lambda trigger route and having it call a flask server that will handle the stream data. Any thoughts on that? — Peter Tao
– Peter Tao, Commented Nov 28, 2018 at 22:54
@PeterTao It's a good idea to use lambda with dynamo streams, but I would recommend you to just get the data, and send it via sqs. — Rafael Marques
– Rafael Marques, Commented Feb 7, 2020 at 13:01

aherve · Accepted Answer · 2021-08-08 08:38:25Z

1

I would still suggest to use lambda. The setup is very easy as well as very robust (it's easy to manage retries,batching, downtimes...)

Then from your lambda invocation you could easily send your data in a convenient way to your existing program (including, but not limited to: SNS, SQS, a custom server webhook, sending the data to a custom pub/sub service you own...etc)

answered Aug 8, 2021 at 8:38

aherve

4,1406 gold badges33 silver badges44 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Dynamodb streams in python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related