Update to_gbq and read_gbq to pandas-gbq 0.5.0 #21628

tswast · 2018-06-25T18:03:15Z

Closes googleapis/python-bigquery-pandas#177

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

I've also verified that the docs build and render well with

python doc/make.py --single read_gbq
python doc/make.py --single DataFrame.to_gbq

Output from scripts/validate_docstrings.py pandas.read_gbq:


################################################################################
######################### Docstring (pandas.read_gbq)  #########################
################################################################################

Load data from Google BigQuery.

This function requires the `pandas-gbq package
<https://pandas-gbq.readthedocs.io>`__.

See the `How to authenticate with Google BigQuery
<https://pandas-gbq.readthedocs.io/en/latest/howto/authentication.html>`__
guide for authentication instructions.

Parameters
----------
query : str
    SQL-Like Query to return data values.
project_id : str, optional
    Google BigQuery Account project ID. Optional when available from
    the environment.
index_col : str, optional
    Name of result column to use for index in results DataFrame.
col_order : list(str), optional
    List of BigQuery column names in the desired order for results
    DataFrame.
reauth : boolean, default False
    Force Google BigQuery to re-authenticate the user. This is useful
    if multiple accounts are used.
private_key : str, optional
    Service account private key in JSON format. Can be file path
    or string contents. This is useful for remote server
    authentication (eg. Jupyter/IPython notebook on remote host).
auth_local_webserver : boolean, default False
    Use the `local webserver flow`_ instead of the `console flow`_
    when getting user credentials.

    .. _local webserver flow:
        http://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_local_server
    .. _console flow:
        http://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_console

    *New in version 0.2.0 of pandas-gbq*.
dialect : str, default 'legacy'
    SQL syntax dialect to use. Value can be one of:

    ``'legacy'``
        Use BigQuery's legacy SQL dialect. For more information see
        `BigQuery Legacy SQL Reference
        <https://cloud.google.com/bigquery/docs/reference/legacy-sql>`__.
    ``'standard'``
        Use BigQuery's standard SQL, which is
        compliant with the SQL 2011 standard. For more information
        see `BigQuery Standard SQL Reference
        <https://cloud.google.com/bigquery/docs/reference/standard-sql/>`__.
location : str, optional
    Location where the query job should run. See the `BigQuery locations
    documentation
    <https://cloud.google.com/bigquery/docs/dataset-locations>`__ for a
    list of available locations. The location must match that of any
    datasets used in the query.

    *New in version 0.5.0 of pandas-gbq*.
configuration : dict, optional
    Query config parameters for job processing.
    For example:

        configuration = {'query': {'useQueryCache': False}}

    For more information see `BigQuery REST API Reference
    <https://cloud.google.com/bigquery/docs/reference/rest/v2/jobs#configuration.query>`__.
verbose : None, deprecated
    Deprecated in Pandas-GBQ 0.4.0. Use the `logging module
    to adjust verbosity instead
    <https://pandas-gbq.readthedocs.io/en/latest/intro.html#logging>`__.

Returns
-------
df: DataFrame
    DataFrame representing results of query.

See Also
--------
pandas_gbq.read_gbq : This function in the pandas-gbq library.
pandas.DataFrame.to_gbq : Write a DataFrame to Google BigQuery.

################################################################################
################################## Validation ##################################
################################################################################

Errors found:
	No examples section found

Output from scripts/validate_docstrings.py pandas.DataFrame.to_gbq:

################################################################################
##################### Docstring (pandas.DataFrame.to_gbq)  #####################
################################################################################

Write a DataFrame to a Google BigQuery table.

This function requires the `pandas-gbq package
<https://pandas-gbq.readthedocs.io>`__.

See the `How to authenticate with Google BigQuery
<https://pandas-gbq.readthedocs.io/en/latest/howto/authentication.html>`__
guide for authentication instructions.

Parameters
----------
destination_table : str
    Name of table to be written, in the form ``dataset.tablename``.
project_id : str, optional
    Google BigQuery Account project ID. Optional when available from
    the environment.
chunksize : int, optional
    Number of rows to be inserted in each chunk from the dataframe.
    Use ``None`` to load the dataframe in a single chunk.
reauth : bool, default False
    Force Google BigQuery to re-authenticate the user. This is useful
    if multiple accounts are used.
if_exists : str, default 'fail'
    Behavior when the destination table exists. Value can be one of:

    ``'fail'``
        If table exists, do nothing.
    ``'replace'``
        If table exists, drop it, recreate it, and insert data.
    ``'append'``
        If table exists, insert data. Create if does not exist.
private_key : str, optional
    Service account private key in JSON format. Can be file path
    or string contents. This is useful for remote server
    authentication (eg. Jupyter/IPython notebook on remote host).
auth_local_webserver : bool, default False
    Use the `local webserver flow`_ instead of the `console flow`_
    when getting user credentials.

    .. _local webserver flow:
        http://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_local_server
    .. _console flow:
        http://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_console

    *New in version 0.2.0 of pandas-gbq*.
table_schema : list of dicts, optional
    List of BigQuery table fields to which according DataFrame
    columns conform to, e.g. ``[{'name': 'col1', 'type':
    'STRING'},...]``. If schema is not provided, it will be
    generated according to dtypes of DataFrame columns. See
    BigQuery API documentation on available names of a field.

    *New in version 0.3.1 of pandas-gbq*.
location : str, optional
    Location where the load job should run. See the `BigQuery locations
    documentation
    <https://cloud.google.com/bigquery/docs/dataset-locations>`__ for a
    list of available locations. The location must match that of the
    target dataset.

    *New in version 0.5.0 of pandas-gbq*.
progress_bar : bool, default True
    Use the library `tqdm` to show the progress bar for the upload,
    chunk by chunk.

    *New in version 0.5.0 of pandas-gbq*.
verbose : bool, deprecated
    Deprecated in Pandas-GBQ 0.4.0. Use the `logging module
    to adjust verbosity instead
    <https://pandas-gbq.readthedocs.io/en/latest/intro.html#logging>`__.

See Also
--------
pandas_gbq.to_gbq : This function in the pandas-gbq library.
pandas.read_gbq : Read a DataFrame from Google BigQuery.

################################################################################
################################## Validation ##################################
################################################################################

Errors found:
	No returns section found
	No examples section found

The docs needs some corrections in order to pass the Pandas docs linter in pandas-dev/pandas#21628

codecov · 2018-06-25T21:24:49Z

Codecov Report

Merging #21628 into master will not change coverage.
The diff coverage is 100%.

@@           Coverage Diff           @@
##           master   #21628   +/-   ##
=======================================
  Coverage    91.9%    91.9%           
=======================================
  Files         154      154           
  Lines       49562    49562           
=======================================
  Hits        45549    45549           
  Misses       4013     4013

Flag	Coverage Δ
#multiple	`90.3% <100%> (ø)`	⬆️
#single	`41.85% <100%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/gbq.py	`25% <100%> (ø)`	⬆️
pandas/core/frame.py	`97.19% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36422a8...7330463. Read the comment docs.

tswast · 2018-06-25T22:04:21Z

Travis failure appears to be unrelated to this change.

jreback · 2018-06-25T22:13:52Z

pandas/io/gbq.py

+        Use the `local webserver flow`_ instead of the `console flow`_
+        when getting user credentials.
+
+        .. _local webserver flow:


i think these break linting, need a #noqa I think

Adding # noqa breaks the link. Sphinx generates a link like pandas/doc/build/html/generated_single/pandas.read_gbq.html#noqahttp://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_local_server when I add this.

I do not get any lint errors when I run git diff upstream/master -u -- "*.py" | flake8 --diff and scripts/validate_docstrings.py pandas.read_gbq also passes.

jreback · 2018-06-25T22:14:23Z

pandas/core/frame.py

-            Name of table to be written, in the form 'dataset.tablename'.
-        project_id : str
-            Google BigQuery Account project ID.
+            Name of table to be written, in the form ``dataset.tablename``.


I think needs to be single backticks

Two backticks is code font in Sphinx RST, which is what I want.

jreback · 2018-06-25T22:15:13Z

doc/source/whatsnew/v0.24.0.txt

@@ -18,6 +18,8 @@ Other Enhancements
 - :func:`Series.mode` and :func:`DataFrame.mode` now support the ``dropna`` parameter which can be used to specify whether NaN/NaT values should be considered (:issue:`17534`)
 - :func:`to_csv` now supports ``compression`` keyword when a file handle is passed. (:issue:`21227`)
 - :meth:`Index.droplevel` is now implemented also for flat indexes, for compatibility with MultiIndex (:issue:`21115`)
+- :func:`to_gbq` and :func:`read_gbq` signature and documentation updated to


can you add here a link to the pandas-gbq docs whatsnew for 0.5.0?

Added link to https://pandas-gbq.readthedocs.io/en/latest/changelog.html#changelog-0-5-0

* Add link to Pandas-GBQ 0.5.0 in what's new. * Remove unnecessary sleep in GBQ tests. Closes googleapis/python-bigquery-pandas#177 Closes pandas-dev#21627

tswast

Updated with link to Pandas-GBQ changelog and rebase on master.

tswast · 2018-06-26T16:10:33Z

pandas/core/frame.py

-            Name of table to be written, in the form 'dataset.tablename'.
-        project_id : str
-            Google BigQuery Account project ID.
+            Name of table to be written, in the form ``dataset.tablename``.


Two backticks is code font in Sphinx RST, which is what I want.

tswast · 2018-06-26T16:33:18Z

doc/source/whatsnew/v0.24.0.txt

@@ -18,6 +18,8 @@ Other Enhancements
 - :func:`Series.mode` and :func:`DataFrame.mode` now support the ``dropna`` parameter which can be used to specify whether NaN/NaT values should be considered (:issue:`17534`)
 - :func:`to_csv` now supports ``compression`` keyword when a file handle is passed. (:issue:`21227`)
 - :meth:`Index.droplevel` is now implemented also for flat indexes, for compatibility with MultiIndex (:issue:`21115`)
+- :func:`to_gbq` and :func:`read_gbq` signature and documentation updated to


Added link to https://pandas-gbq.readthedocs.io/en/latest/changelog.html#changelog-0-5-0

tswast · 2018-06-26T16:39:08Z

pandas/io/gbq.py

+        Use the `local webserver flow`_ instead of the `console flow`_
+        when getting user credentials.
+
+        .. _local webserver flow:


Adding # noqa breaks the link. Sphinx generates a link like pandas/doc/build/html/generated_single/pandas.read_gbq.html#noqahttp://google-auth-oauthlib.readthedocs.io/en/latest/reference/google_auth_oauthlib.flow.html#google_auth_oauthlib.flow.InstalledAppFlow.run_local_server when I add this.

I do not get any lint errors when I run git diff upstream/master -u -- "*.py" | flake8 --diff and scripts/validate_docstrings.py pandas.read_gbq also passes.

tswast · 2018-06-26T20:14:57Z

Looks like Travis and Circle are both happy now that I rebased.

jorisvandenbossche · 2018-06-26T20:27:59Z

I am not really familiar with the gbq code, but, did we consider making this just a passthrough of *args, **kwargs ? Then it would automatically work with the installed pandas_gbq

tswast · 2018-06-26T20:33:49Z

@jorisvandenbossche I considered (and even coded) that in #20564

The development experience is nicer (IDEs understand them better) without using **kwargs. It is somewhat rare to add new arguments to Pandas-GBQ, though I do have a couple changes in mind to streamline auth googleapis/python-bigquery-pandas#161.

The location field is special because it's a new field on the BigQuery API as opposed to a new field in the BigQuery API resources.

jreback · 2018-06-26T21:12:19Z

i think writing out the kwargs is nice from a usability pov - we do this for other apis as well

jorisvandenbossche · 2018-06-26T21:29:25Z

Yep, no problem, was just wondering!

jreback · 2018-06-26T22:26:05Z

thanks @tswast !

* Add link to Pandas-GBQ 0.5.0 in what's new. * Remove unnecessary sleep in GBQ tests. Closes googleapis/python-bigquery-pandas#177 Closes pandas-dev#21627

tswast added a commit to tswast/python-bigquery-pandas that referenced this pull request Jun 25, 2018

DOC: synchronize docs changes with pandas

4db5d0d

The docs needs some corrections in order to pass the Pandas docs linter in pandas-dev/pandas#21628

tswast mentioned this pull request Jun 25, 2018

DOC: synchronize docs changes with pandas googleapis/python-bigquery-pandas#190

Merged

tswast added a commit to googleapis/python-bigquery-pandas that referenced this pull request Jun 25, 2018

DOC: synchronize docs changes with pandas (#190)

06ff699

The docs needs some corrections in order to pass the Pandas docs linter in pandas-dev/pandas#21628

gfyoung added IO Google Enhancement labels Jun 25, 2018

jreback requested changes Jun 25, 2018

View reviewed changes

tswast force-pushed the master branch from dbec900 to 9a470b3 Compare June 26, 2018 16:46

ENH: Update to_gbq and read_gbq to pandas-gbq 0.5.0

7330463

* Add link to Pandas-GBQ 0.5.0 in what's new. * Remove unnecessary sleep in GBQ tests. Closes googleapis/python-bigquery-pandas#177 Closes pandas-dev#21627

tswast force-pushed the master branch from 7bcf910 to 7330463 Compare June 26, 2018 16:49

tswast commented Jun 26, 2018

View reviewed changes

tswast mentioned this pull request Jun 26, 2018

Add anchor links to versions in the changelog googleapis/python-bigquery-pandas#191

Merged

jreback added this to the 0.24.0 milestone Jun 26, 2018

jreback approved these changes Jun 26, 2018

View reviewed changes

jreback merged commit 2a7e1f2 into pandas-dev:master Jun 26, 2018

alimcmaster1 mentioned this pull request Aug 28, 2018

Progress bar gbq #22399

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to_gbq and read_gbq to pandas-gbq 0.5.0 #21628

Update to_gbq and read_gbq to pandas-gbq 0.5.0 #21628

tswast commented Jun 25, 2018

codecov bot commented Jun 25, 2018 •

edited

Loading

tswast commented Jun 25, 2018

jreback Jun 25, 2018

tswast Jun 26, 2018

jreback Jun 25, 2018

tswast Jun 26, 2018

jreback Jun 25, 2018

tswast Jun 26, 2018

tswast left a comment

tswast Jun 26, 2018

tswast Jun 26, 2018

tswast Jun 26, 2018

tswast commented Jun 26, 2018

jorisvandenbossche commented Jun 26, 2018

tswast commented Jun 26, 2018

jreback commented Jun 26, 2018

jorisvandenbossche commented Jun 26, 2018

jreback commented Jun 26, 2018

Update to_gbq and read_gbq to pandas-gbq 0.5.0 #21628

Update to_gbq and read_gbq to pandas-gbq 0.5.0 #21628

Conversation

tswast commented Jun 25, 2018

codecov bot commented Jun 25, 2018 • edited Loading

Codecov Report

tswast commented Jun 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tswast commented Jun 26, 2018

jorisvandenbossche commented Jun 26, 2018

tswast commented Jun 26, 2018

jreback commented Jun 26, 2018

jorisvandenbossche commented Jun 26, 2018

jreback commented Jun 26, 2018

codecov bot commented Jun 25, 2018 •

edited

Loading