AWS DynamoDB retrieve entire table

Question

Folks, Retrieving all items from a DynamoDB table, I would like to replace the scan operation with a query.

Currently I am pulling in all the table's data via the following (python):

drivertable = Table(url['dbname'])
all_drivers = []
all_drivers_query = drivertable.scan()
for x in all_drivers_query:
  all_drivers.append(x['number'])

How would i change this to use the query API?

Thanks!

Why would you want to do that ? Scan() is for full table scans (i.e. retrieve all values) while query() is for getting specific values (hash or hash + range) ... What is your use case ? What do you think query will bring you that scan does not ? — Sébastien Stormacq
– Sébastien Stormacq, Commented Dec 16, 2014 at 13:53

mkobit · Accepted Answer · 2014-12-17 00:33:43Z

3

+50

There is no way to query and get the entire results of the table. As of right now, you have a few options if you want to get all of your data out of a DynamoDB, and all of them involve actually reading the data out of DynamoDB:

Scan the table. It can be done faster with the expense of using much more read capacity by using a parallel scan
Export your data using AWS Data Pipelines. You can configure the export job for where and how it should store your data.
Using one of the AWS event platforms for new data and denormalize it. For all new data you can get a time-ordered stream of all updates to the table from DynamoDB Update Streams or process events using AWS Lambda

answered Dec 17, 2014 at 0:33

mkobit

47.8k12 gold badges162 silver badges159 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Chen Harel · Accepted Answer · 2014-12-17 12:45:41Z

2

You can't query an entire table. Query is used to retrieve a set of items by supplying a hash key (part of the complex primary key hash-range of the table).

One can not use query without knowing the hash keys.

EDIT as a bounty was added to this old question that asks:

How do I get a list of hashes from DynamoDB?

Well - In Dec 2014 you still can't ask via a single API for all hash keys of a table. Even if you go and put a GSI you still can't get a DISTINCT hash count.

The way I would solve this is with de-normalization. Keep another table with no range key and put every hash there together with the main table. This adds house-keeping overhead to your application level (mainly when removing), but solves the problem you asked.

edited Dec 17, 2014 at 12:45

answered Nov 10, 2013 at 22:41

Chen Harel

10.1k5 gold badges48 silver badges60 bronze badges

Collectives™ on Stack Overflow

AWS DynamoDB retrieve entire table

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related