How to export data from Google BigQuery to PostgreSQL

Question

I have a table stored in BigQuery and I would like to create a copy of this table+data within a PostgreSQL database. This PostgreSQL is in Google Cloud SQL.

This export will occur daily, that is, the same table will be exported daily to PostgreSQL, so the ideal solution can be automated/repeated, rather than a manual solution.

How can I achieve this?

The title of this question is incorrect. You want to export a table. There is a huge difference between exporting a table and exporting data. I'm mainly interested in exporting the entire database and found your question as well as the answer useless. — Ash
– Ash, Commented Jan 10, 2024 at 15:46
@Ash The title is not incorrect. Your interpretation is correct. Data != database. Data is a vague mass noun, so could refer to anything, which OP defines in their question. But you wanted more than just 'data' from the outset; you wanted an entire database. That's not OP's fault nor the answerer's fault. The question and answer are clear and correct. — TylerH
– TylerH, Commented Jan 11, 2024 at 16:38

Damião Martins · Accepted Answer · 2022-05-28 17:30:59Z

1

You can use the BigQuery and CloudSQL APIs to do that, exporting the BQ table as CSV to cloud storage and then importing the file as table in CloudSQL. Then you can automate this process using Cloud Functions + Cloud Scheduler (or even Cloud DataFlow for more complex pipelines).

answered May 28, 2022 at 17:30

Damião Martins

1,8691 gold badge12 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Clayton Bonelli - Agriness Over a year ago

I read that Bigquery has the "EXPORT DATA" command which allows you to export data to a file, for example CSV. Do you consider this command as a solution to export data using only SQL syntax? That is, without using Python or another language? On the part of Postgres, I don't know if there is any SQL command from Postgres that would allow reading a CSV file directly from Cloud Storage

TylerH Over a year ago

This answer would be improved by showing, at least in part, how to "use the Big Query and Cloud SQL APIs to do that".

Collectives™ on Stack Overflow

How to export data from Google BigQuery to PostgreSQL

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related