How should a client handle added/removed/split DynamoDB Stream Shards?

Question

Shards are ephemeral: They are created and deleted automatically, as needed. Any shard can also split into multiple new shards; this also occurs automatically. (Note that it is also possible for a parent shard to have just one child shard.) A shard might split in response to high levels of write activity on its parent table, so that applications can process records from multiple shards in parallel.

In my application, I don't really care which shard data comes from, I just want to process all changes. When you set up your stream processors, though, you have to subscribe to a shard or set of shards. How can you be sure you're getting all the changes if shards can be removed/split up/added at any time?

Should a client poll the list of shards for changes?

ketan vijayvargiya · Accepted Answer · 2016-10-09 06:09:41Z

2

Yes. If you are using the low-level APIs of DynamoDB streams, you need to take care of these scenarios on your own.

A better idea is to use the DynamoDB Streams Kinesis Adapter, which allows you to use the Amazon Kinesis Client Library. The latter takes care of all the above-mentioned complexities for you, thereby making your life simpler.

Relevant documentation can be found here.

answered Oct 9, 2016 at 6:09

ketan vijayvargiya

5,6591 gold badge24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How should a client handle added/removed/split DynamoDB Stream Shards?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related