2

Could someone share an example of a job config for uploading json newline_delimited file to a new Bigquery table, please?

Trying to do this based on google docs with no success so far.

1 Answer 1

2

This example from GCP repository is a good one for loading data from GCS.

The only thing you will have to adapt in your code is setting the job.source_format to be the new delimited json file, like so:

def load_data_from_gcs(dataset_name, table_name, source):
    bigquery_client = bigquery.Client()
    dataset = bigquery_client.dataset(dataset_name)
    table = dataset.table(table_name)
    job_name = str(uuid.uuid4())

    job = bigquery_client.load_table_from_storage(
        job_name, table, source)

    job.source_format = 'NEWLINE_DELIMITED_JSON'
    job.begin()

    wait_for_job(job)

    print('Loaded {} rows into {}:{}.'.format(
        job.output_rows, dataset_name, table_name))

(The correct thing would be to receive this parameter as input in your function but this works as an example).

Also, the table should already exist when you run this code (I looked for schema auto-detection in the Python API but it seems there isn't one yet).

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.