pandas.io.gbq Version 2

[jacobschaer]

closes #5840 (as new interface makes it obsolete)
closes #6096

@jreback : We still have some documentation to work on, but we would like your initial thoughts on what we have so far. The key change for this version is the removal of bq.py as a dependency (except as a setup method for a test case). Instead, we rely entirely on the BigQuery python API. We also simplified to_gbq() significantly. Though it cost a few features, the code is much more manageable and less error prone (thanks Google!). Test cases are much more granular and run significantly faster. To use the test cases fully, a BigQuery project_id is still required, though there are some unittests offline.

[jreback]

can u rebase on master?

also flip github switch for Travis

[jacobschaer]

I rebased to master, and Travis was turned on. I'm still working to get Travis passing - mostly just a matter of getting the right dependencies in place and being sure integration tests are skipped.

[jreback]

don;'t you still need the bigquery package so that bq is installed? (or is that in the google-api-python-client), what a horrible package name google!

[jreback]

also if requirements are changing, pls update install.rst as well

[jacobschaer]

bigquery is only required for the to_gbq() test suite, which can't be run in CI anyways due to lack of valid project id. Will update install.rst soon.

[jreback]

this going to close #5840 ?

[jacobschaer]

@jreback : I believe #5840 was no longer an issue due to a backend change with bigquery. However, it's no longer an issue with our code, since we track the page tokens we've seen so far and ensure we don't get a duplicate.

[jacobschaer]

These @unittest.skipIf seem to cause backward-compatibility issues... Perhaps I should switch all these over to whatever the nose equivalent is?

[jreback]

pandas doesn't implement (properly);

make these regular tests and just do a raise nose.SkipTest(slipping message) when u skip
they are indicated by travis at the very end (do u can see that u skipped them)

[jreback]

@jacobschaer you are doing small doc change to close #6096 as well?

[jreback]

@jacobschaer like to get this in soon...how's coming?

[jreback]

@jacobschaer how's this coming?

[jreback]

@jacobschaer pls rebase on master

[jacobschaer]

OK, all rebased and mostly squashed. Were there any documentation things we're missing?

[jorisvandenbossche]

I think read_gbq is already available as a top-level function. So also use it here as such?

[jreback]

@jacobschaer

the 3.4 and numpy-dev builds are failing because they are not installing httplib2 which defacto is a requirement of gbq now. those tests should be skipping (because of the import failure).

those builds need to pass (they are experimental just because the build matrix comes back faster)

[jacobschaer]

@jreback Seems 3.4 and numpy-dev builds now pass.

[jreback]

reference the issue number here (this PR number is ok)

[jreback]

put this in the API changes section as well

[jorisvandenbossche]

or in an 'experimental' (sub)section? (as it was and still is tagged as experimental?)

[jreback]

yes...put that in the experimental section (down a little on the page in v0.14.1.txt)

[azbones]

Moved this to the experimental section and added issue number with this PR number.

[jreback]

@jacobschaer pls squash down to a small number of commits (and remove merge branches), and rebase on master

[jreback]

@jacobschaer can you rebase and squash?

[jacobschaer]

All rebased and squashed.

[jreback]

@jacobschaer thanks for this!

pls review the docs when they are built in case a followup is needed. http://pandas-docs.github.io/pandas-docs-travis/

[jreback]

@jacobschaer If you have a chance. Can you break the big paragraph into say bullet points:

Finally, you can append data to a BigQuery table from a pandas DataFrame using the to_gbq() 
function. This function uses the Google streaming API which requires that your destination table exists in
 BigQuery. Given the BigQuery table already exists, your DataFrame should match the destination table 
in column order, structure, and data types. DataFrame indexes are not supported. By default, rows are 
streamed to BigQuery in chunks of 10,000 rows, but you can pass other chuck values via the chunksize 
argument. You can also see the progess of your post via the verbose flag which defaults to True. The 
http response code of Google BigQuery can be successful (200) even if the append failed. For this 
reason, if there is a failure to append to the table, the complete error response from BigQuery is 
returned which can be quite long given it provides a status for each row. You may want to start with 
smaller chuncks to test that the size and types of your dataframe match your destination table to make 
debugging simpler.

http://pandas-docs.github.io/pandas-docs-travis/io.html#io-bigquery