How do I efficiently migrate the BigQuery Tables to On-Prem Postgres?

Question

I need to migrate the tables from the BigQuery to the on-prem Postgres database.

How can I efficiently achieve that?

Some thoughts that are coming

I will use Google APIs to export the data from the tables
Store it locally
And finally, import to Postgres

But I am not sure if that can be done for a huge amount of data in TBs. Also, how can I automate this process? Can I use Jenkins for that?

Tlaquetzal · Accepted Answer · 2019-11-19 15:01:35Z

1

Exporting the data from BigQuery, store it and importing it to PostgreSQL is a good approach. Here are other two alternatives that you can consider:

1) There's a PostgreSQL wrapper for BigQuery that allows to query directly from BigQuery. Depending on your case scenario this might be the easiest way to transfer the data; although, for TBs it might not be the best approach. This suggestion was made by @David in this SO question.

2) Using Dataflow. You can create a ETL process using Apache Beam to made the transfer. Take a look at this how-to for transferring data from BigQuery to CloudSQL. You would need to adapt it for local PostgreSQL, but the idea maintains.

Here's another SO answer that gives more context on this approach.

answered Nov 19, 2019 at 15:01

Tlaquetzal

2,8501 gold badge15 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How do I efficiently migrate the BigQuery Tables to On-Prem Postgres?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related