Export DynamoDB Table as JSON to S3 Using Lambda Function

Question

I am trying to export DynamoDB table as JSON format to S3 and from there import it to BigQuery. The important part is exporting DynamoDB table as JSON format to S3, because the table I am working on is not a small table. This table contains 5.6 million records and about 15.000 (on a quiet day) new records are inserted every day. I came across with a blog post which suggests Lambda (ref: http://randomwits.com/blog/export-dynamodb-s3) function but table.scan() function does not work well with large tables.

So how can I export DynamoDB table in JSON format to S3 and from there import it to BigQuery efficiently? I saw some options like HEVO, Glue, etc. But I don't know which way would be the most efficient.

DynamoDB has a new feature "export to S3" which offers a good solution but in that case I will have to enable Point-in-time recovery (PITR) for Amazon DynamoDB. I am unsure if it will worth it, if the solution will be effective. — PurpleGreen
– PurpleGreen, Commented Nov 18, 2020 at 9:43
I would enable PITR, personally. Related: stackoverflow.com/questions/18896329/export-data-from-dynamodb and stackoverflow.com/questions/33357821/…. — jarmod
– jarmod, Commented Nov 18, 2020 at 12:10
@M.EceErcan Please go through the below link. aws.amazon.com/blogs/aws/… — Sarathy Velmurugan
– Sarathy Velmurugan, Commented Feb 17, 2021 at 6:25

Levi · Accepted Answer · 2020-11-18 11:14:35Z

1

You can do this with AWS lambda, lambda is triggered by DynamoDB stream, then this lambda will write to cloud logging, from cloud logging you will have to create a sink and make big query as the destination

answered Nov 18, 2020 at 11:14

Levi

997 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jarmod Over a year ago

That will help export new data, but not the existing data.

Levi Over a year ago

Then you can utilize dynamodb export to s3, then query the data using athena, the query results can be put on a new bucket -> AWS Lambda -> Cloud Logging -> Sink to BQ docs.aws.amazon.com/amazondynamodb/latest/developerguide/…

jarmod Over a year ago

Right, I'm just pointing out that your answer addresses the change data capture, but not the original data.

Collectives™ on Stack Overflow

Export DynamoDB Table as JSON to S3 Using Lambda Function

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related