python2.7 throwing error when calling bigquery api

Question

I am using google-api-python-client for inserting a json record to bigquery and when I try to unittest the method using python unittest, I am getting error in exactly this line

The code is as follows:

def write_to_bigquery(self, timeseries, metadata):
        response = {}
        json_msg_list = []
        stats = {}
        if not timeseries or "points" not in timeseries:
            logging.debug("No timeseries data to write to BigQuery")
            msgs_written = 0
            metadata["msg_without_timeseries"] = 1
            error_msg_cnt = 0
        else:   
            rows = build_rows(timeseries, metadata)
            print("rows", rows) //This gets printed
            bigquery = build('bigquery', 'v2', cache_discovery=False) 
            print("after rows", rows) //Control does not reach here
body = {
            "kind": "bigquery#tableDataInsertAllRequest",
            "skipInvalidRows": "false",
            "rows": json_row_list
        }
        logging.debug('body: {}'.format(json.dumps(body, sort_keys=True, indent=4)))

        response = bigquery.tabledata().insertAll(
            projectId=app_identity.get_application_id(),
            datasetId=config.BIGQUERY_DATASET,
            tableId=config.BIGQUERY_STATS_TABLE,
            body=body
        ).execute()
        logging.debug("BigQuery said... = {}".format(response))

and this is the error I get

     Traceback (most recent call last):
  File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "main.py", line 422, in post
 File "/home/barumugham/.local/lib/python2.7/site-packages/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "main.py", line 422, in post
    self.write_to_bigquery(data, metadata)
  File "main.py", line 296, in write_to_bigquery
    bigquery = build('bigquery', 'v2', cache_discovery=False)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/discovery.py", line 258, in build
    adc_key_path=adc_key_path,
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/discovery.py", line 423, in build_from_document
    credentials = _auth.default_credentials()
  File "/usr/local/lib/python2.7/dist-packages/googleapiclient/_auth.py", line 44, in default_credentials
    credentials, project_id = checker()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/_default.py", line 186, in _get_gae_credentials
    project_id = app_engine.get_project_id()
  File "/usr/local/lib/python2.7/dist-packages/google/auth/app_engine.py", line 77, in get_project_id
    return app_identity.get_application_id()
  File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/app_identity/app_identity.py", line 455, in get_application_id
    _, domain_name, display_app_id = _ParseFullAppId(full_app_id)
  File "/usr/lib/google-cloud-sdk/platform/google_appengine/google/appengine/api/app_identity/app_identity.py", line 436, in _ParseFullAppId
    psep = app_id.find(_PARTITION_SEPARATOR)
AttributeError: 'NoneType' object has no attribute 'find'

I am new to python and bigquery so any help is appreciated thanks

Its might be a credentials problem. How are you authenticating? — rmesteves
– rmesteves, Commented Jun 22, 2020 at 7:02
I am not sure how to authenticate, I am running these as unittests from cloudshell command line and I want to invoke the biquery dataset for inserting data after building it and not passing any credentials — user1403505
– user1403505, Commented Jun 22, 2020 at 10:13
Do you have GOOGLE_APPLICATION_CREDENTIALS set in your environment or something like that or you dont have any authentication at all? — rmesteves
– rmesteves, Commented Jun 22, 2020 at 10:46

rmesteves · Accepted Answer · 2020-06-22 13:14:35Z

1

I would recommend you using the BigQuery Python SDK

For that, you first need to install it in you Python. You can do that by running:

pip install google-cloud-bigquery

After that you use a code like this insert json records to your table:

from google.cloud import bigquery

# Construct a BigQuery client object.
client = bigquery.Client()

table_id = "project_id.dataset.table"

# Your JSON keys must correspond to your table column names
json_list = [{"your": "json", "data":"here"},{"your": "json", "data":"here"},{"your": "json", "data":"here"}, ...]

# Get table reference
table = client.get_table(table_id)
rows_to_insert = json_list
# Insert the data into your table
errors = client.insert_rows(table, rows_to_insert)

Finally, I'd like to say that Python 2 is considered deprecated already. If possible, update it to Python 3

answered Jun 22, 2020 at 13:14

rmesteves

4,1359 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python2.7 throwing error when calling bigquery api

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related