0

I am reading data from a CSV file, inserting the data to a Big Query table using insertAll() method from Streaming Insert as shown below:

InsertAllResponse response = dfsf.insertAll(InsertAllRequest.newBuilder(tableId).setRows(rows).build());

rows here is an Iterable declared like this:

Iterable<InsertAllRequest.RowToInsert> rows

Now, I am actually batching the rows to insert into a size of 500 as suggested here - link to suggestion

After all the data has been inserted, how do I count the total number of rows that were inserted? I want to find that out and log it to log4j.

1 Answer 1

2

This can be done one of two ways

  1. The BigQuery Jobs API via the getQueryResults https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs/getQueryResults
  2. Cloud Logging, the output you want in the tableDataChange field.

Here is a sample output:

{
  "protoPayload": {
    "@type": "type.googleapis.com/google.cloud.audit.AuditLog",
    "status": {},
    "authenticationInfo": {
      "principalEmail": "service_account"
    },
    "requestMetadata": {
      "callerIp": "2600:1900:2000:1b:400::27",
      "callerSuppliedUserAgent": "gl-python/3.7.1 grpc/1.22.0 gax/1.14.2 gapic/1.12.1 gccl/1.12.1,gzip(gfe)"
    },
    "serviceName": "bigquery.googleapis.com",
    "methodName": "google.cloud.bigquery.v2.JobService.InsertJob",
    "authorizationInfo": [
      {
        "resource": "projects/project_id/datasets/dataset/tables/table",
        "permission": "bigquery.tables.updateData",
        "granted": true
      }
    ],
    "resourceName": "projects/project_id/datasets/dataset/tables/table",
    "metadata": {
      "tableDataChange": {
        "deletedRowsCount": "2",
        "insertedRowsCount": "2",
        "reason": "QUERY",
        "jobName": "projects/PRJOECT_ID/jobs/85f19bdd-aff5-4abe-9283-9f0bc9ed3ce8"
      },
      "@type": "type.googleapis.com/google.cloud.audit.BigQueryAuditMetadata"
    }
  },
  "insertId": "7x7ye390qm",
  "resource": {
    "type": "bigquery_dataset",
    "labels": {
      "project_id": "PRJOECT_ID",
      "dataset_id": "dataset-id"
    }
  },
  "timestamp": "2020-10-26T07:00:22.960735Z",
  "severity": "INFO",
  "logName": "projects/PRJOECT_ID/logs/cloudaudit.googleapis.com%2Fdata_access",
  "receiveTimestamp": "2020-10-26T07:00:23.763159336Z"
}
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.