Authenticate with credentials

[rxuniverse]

Provide an option to authenticate with credentials instead of private keys.

Example use case: On Google Cloud compute engine, I can obtain credentials in a compute engine instance based on the service account assigned to compute engine, but I can not directly get the private key.

If I pass credentials in the private key field, it fails as expected.
>>> from google.auth import compute_engine
>>> credentials = compute_engine.Credentials()
>>> import pandas
>>> pandas.io.gbq.read_gbq(sql, project, private_key=credentials)
/env/local/lib/python2.7/site-packages/pandas_gbq/gbq.py:798: FutureWarning: verbose is deprecated and will be removed in a future version. Set logging level in order to vary verbosity "verbosity", FutureWarning, stacklevel=1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/env/local/lib/python2.7/site-packages/pandas/io/gbq.py", line 99, in read_gbq **kwargs) File "/env/local/lib/python2.7/site-packages/pandas_gbq/gbq.py", line 810, in read_gbq dialect=dialect, auth_local_webserver=auth_local_webserver) File "/env/local/lib/python2.7/site-packages/pandas_gbq/gbq.py", line 180, in __init__ self.credentials = self.get_credentials() File "/env/local/lib/python2.7/site-packages/pandas_gbq/gbq.py", line 189, in get_credentials return self.get_service_account_credentials() File "/env/local/lib/python2.7/site-packages/pandas_gbq/gbq.py", line 409, in get_service_account_credentials "Private key is missing or invalid. It should be service " pandas_gbq.gbq.InvalidPrivateKeyFormat: Private key is missing or invalid. It should be service account private key JSON (file path or string contents) with at least two keys: 'client_email' and 'private_key'. Can be obtained from: https://console.developers.google.com/permissions/serviceaccounts

[max-sixty]

Can you confirm that it doesn't work if you pass in no details? Application Default Credentials (which are supplied in GCE) should work by default.

[rxuniverse]

To clarify, I am running this in a pod in Kubernetes Engine. I don't think there is an Application Default Credentials there.

I am not sure if there is an Application Default Credentials for compute engine... I saw the documentation grouped Compute Engine and Kubernetes Engine together, so I assumed the situations are the same...

I did try to run query without the private key field (from a shell in a pod in Kubernetes Engine), it asks for authorization
>>> pandas.io.gbq.read_gbq(sql, project)
Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=495642085510-k0tmvj2m941jhre2nbqka17vqpjfddtd.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fbigquery&state=I9HkEQv3ywVTfJYfH1q1bIrlQLDrVl&prompt=consent&access_type=offline Enter the authorization code:

[max-sixty]

I'm fairly confident that there are ADCs on GKE. What are you seeing in the docs that makes you think there isn't?

If your application runs on Compute Engine, Kubernetes Engine, the App Engine flexible environment, or Cloud Functions, you don't need to create your own service account. Compute Engine includes a default service account that is automatically created for you

What do you see if you run import google; google.auth.default()?

[tswast]

Last I tried there was some setup required to get application default credentials to work on Kubernetes Engine.

[tswast]

I agree that a way to provide general credentials is desired and have created a design proposal at #161. Closing this issue as a duplicate of that proposal.

[rxuniverse]

@maxim-lian I tried it, it returns a credentials and project name string pair. No private keys though. the env variable APPLICATION_DEFAULT_CREDENTIALS is also not set by default.

@tswast this is exactly the way I am doing it right now, creating a credentials, put it in a volume and mount it to a Kubernetes container, then set the APPLICATION_DEFAULT_CREDENTIALS to point to that credentials.

#161 looks very comprehensive! Feel free to ping me to discuss or verify CLs.