Modeling data in NoSQL DynamoDB

Question

I'm trying to figure out how to model the following data in AWS DynamoDB table.

I have a lot of IOT devices, each sends telemetry data every few seconds.

Attributes

device_id
timestamp
malware_name
company_name
action_performed (two possible values)

Queries

Show all incidents that happened in the last week.
Show all incidents for a specific device_id.
Show all incidents with action "unable_to_remove".
show all incidents related to specific malware.
Show all incidents related to specific company.

Thoughts

I understand that I can add GSI's for each attribute, but I would like to use GSI's only if there is no other choice as it costs me more money.
What would be the main primary-key (partition-key:sort-key) ?

Please share you thoughts, I care about them more than I care about the perfect answer as I'm trying to learn how to think and what to consider instead of having an answer for a specific question.

Thanks a lot !

You're best start is reading about how to store time series data in your DB, try this part of AWS documentation and look for more references to this kind of DB design in DynamoDB - docs.aws.amazon.com/amazondynamodb/latest/developerguide/… — Yaron Idan
– Yaron Idan, Commented Jun 11, 2016 at 12:24

ketan vijayvargiya · Accepted Answer · 2016-06-12 00:59:13Z

2

If you absolutely need the querability patterns mentioned, you have no way out but create GSIs for each. That too has its set of caveats:

For query #1, your GSI would be incident_date (or whatever) as partition-key and device_id as sort-key. This might lead to hot partitioning in DynamoDB, based on your access patterns.
There is a limit of 5 GSIs per table, that you'll use up right away. What'll you do if you need to support another kind of query in future?

While evaluating pros and cons of using NoSQL for a given situation, one needs to consider both read and write access patterns. So, the question you should ask is, why DynamoDB?

For e.g., do you really need realtime queries? If not, you can use DynamoDB as the main database and periodically sync data (using AWS Lambda or Kinesis Firehose) to EMR or Redshift for later batch processing.

Edit: Proposed primary key:

device_id as partition-key and incident_date as sort-key, if you know that no 2 or more incidents, for a given device_id, can come at exact same time.
If above doesn't work, then incident_id as partition-key and incident_date as sort-key.

edited Jun 12, 2016 at 0:59

answered Jun 11, 2016 at 12:47

ketan vijayvargiya

5,6591 gold badge24 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

johni Over a year ago

What would be the primary key in your way?

Asaf Savich Over a year ago

So @ketan, you would suggest using those as primary and sort key and use another 4 GSIs?

ketan vijayvargiya Over a year ago

Yes. If you need those 5 query patterns in real-time, you have no other choice

johni Over a year ago

How would I retrieve all incidents from last week in your modeling? Btw, I do not have incident_id

Collectives™ on Stack Overflow

Modeling data in NoSQL DynamoDB

Attributes

Queries

Thoughts

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Attributes

Queries

Thoughts

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related