Getting all rows from an Amazon DynamoDB table

Question

I need to get all rows from my DynamoDB table. There are approximately 500k records and every time I call scan method from Java SDK DynamoDBMapper I get different number of rows. So I tried to use scanPage method and use lastEvaluationKey from ScanResultPage object for pagination (because I think query result set size limit is 1MB) thinking it will work. But I get same result. I did something like this:

        List<String> ids = new ArrayList<>();
        DynamoDBScanExpression dynamoDBScanExpression = new DynamoDBScanExpression();
        do {
            ScanResultPage<EntityObject> scanResultPage = dynamoDBMapper.scanPage(EntityObject.class,
                    dynamoDBScanExpression);
            scanResultPage.getResults().forEach(result -> ids.add(result.getId()));
            log.info("Last evaluated key: {}", scanResultPage.getLastEvaluatedKey());
            dynamoDBScanExpression.setExclusiveStartKey(scanResultPage.getLastEvaluatedKey());
        } while (dynamoDBScanExpression.getExclusiveStartKey() != null);

What can I do to actually get all rows?

smac2020 · Accepted Answer · 2022-03-14 16:44:15Z

3

You are using a very old API and this API is not best practice. Amazon recommends the AWS SDK for Java V2.

For your use case, take a look at Retrieving paginated results using the AWS SDK for Java 2.x. This topic talks about how to handle AWS operations that return paginated results when the response object is too large to return in a single response. Details here:

Retrieving paginated results using the AWS SDK for Java 2.x

HEre is an example that demonstrates how to use auto pagination with the asynchronous DynamoDB client

https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/javav2/example_code/dynamodbasync/src/main/java/com/example/dynamodbasync/AsyncPagination.java

In the AWS SDK for Java 1.0, the response contained a token you had to use to retrieve the next page of results. New in the AWS SDK for Java 2.x are autopagination methods that make multiple service calls to get the next page of results for you.

edited Mar 14, 2022 at 16:44

answered Mar 14, 2022 at 16:09

smac2020

11k4 gold badges27 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Getting all rows from an Amazon DynamoDB table

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related