Google Business Profile: Request had insufficient authentication scopes.

[jamesmacwhite]

Describe the bug

I'm recently having an issue with the Google Business Profile OAuth scope https://www.googleapis.com/auth/business.manage with the Google Provider. It has been working fine through Consume for a long while. More recently due to 2FA being applied on the account being used for OAuth is caused the token to become invalid. After reconnecting and getting a new token everything is fine but the token is not automatically refreshing as requests are then failing after about 1 hour, which seems like token expiry.

The OAuth consent is set to internal only. This has worked fine, it just means that only accounts under our domain can use it, which is fine. I did also try making it an external application and authenticate under a general Google account that isn't under our Google Workspace but it didn't seem to do anything.

I am calling the authenticated client with Consume::$plugin->getService()->fetchData() which as far as I know should handle token refreshing?

I dumped the OAuth authorization event to confirm what was being passed and it looked OK in terms of handling offline modes for token refreshing.

I don't think there's been any major changes to the Google provider lately, so it is a bit odd.

Checking the auth_oauth_tokens table, there is a refresh token present.

Steps to reproduce

1. Configure Google Provider with Google Business Profile scope
2. Authenticate to get a token
3. The token will expire and not refresh

Craft CMS version

5.5.6.1

Plugin version

2.0.1

Multi-site?

No

Additional context

No response

[engram-design]

So yep, Auth will handle the authenticated client, including refreshing any tokens. That includes when calling the Consume fetchData() which is going to use the authenticated client.

Now I would ask if there was a refresh token, but you've noted that there is, so that's a good first-call.

I wouldn't have thought the https://www.googleapis.com/auth/business.manage scope makes a difference, and while I'm just going to wait a few hours to test that, other scopes aren't affecting my client.

Anything in your logs?

[jamesmacwhite]

Nothing in the logs other than the 403 error after 1 hour. Configuring a new token works for an hour until it expires again.

[jamesmacwhite]

{
"error": {
"code": 403,
"message": "Request had insufficient authentication scopes.",
"status": "PERMISS (truncated...)
” /var/www/html/vendor/guzzlehttp/guzzle/src/Exception/RequestException.php:111. Trace: “#0 /var/www/html/vendor/guzzlehttp/guzzle/src/Middleware.php(72): GuzzleHttp\Exception\RequestException::create()
#1 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(209): GuzzleHttp\Middleware::GuzzleHttp{closure}()
#2 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(158): GuzzleHttp\Promise\Promise::callHandler()
#3 /var/www/html/vendor/guzzlehttp/promises/src/TaskQueue.php(52): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise{closure}()
#4 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(251): GuzzleHttp\Promise\TaskQueue->run()
#5 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(227): GuzzleHttp\Promise\Promise->invokeWaitFn()
#6 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(272): GuzzleHttp\Promise\Promise->waitIfPending()
#7 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(229): GuzzleHttp\Promise\Promise->invokeWaitList()
#8 /var/www/html/vendor/guzzlehttp/promises/src/Promise.php(69): GuzzleHttp\Promise\Promise->waitIfPending()
#9 /var/www/html/vendor/guzzlehttp/guzzle/src/Client.php(124): GuzzleHttp\Promise\Promise->wait()
#10 /var/www/html/vendor/league/oauth2-client/src/Provider/AbstractProvider.php(715): GuzzleHttp\Client->send()
#11 /var/www/html/vendor/verbb/auth/src/base/ProviderTrait.php(160): League\OAuth2\Client\Provider\AbstractProvider->getResponse()
#12 /var/www/html/vendor/verbb/auth/src/base/OAuthProviderTrait.php(249): verbb\auth\providers\Google->getApiRequest()
#13 /var/www/html/vendor/verbb/consume/src/services/Service.php(145): verbb\consume\base\OAuthClient->request()
#14 /var/www/html/vendor/verbb/consume/src/services/Service.php(76): verbb\consume\services\Service->fetchRawData()
#15 [internal function]: verbb\consume\services\Service->verbb\consume\services{closure}()
#16 /var/www/html/vendor/yiisoft/yii2/caching/Cache.php(608): call_user_func()
#17 /var/www/html/vendor/verbb/consume/src/services/Service.php(74): yii\caching\Cache->getOrSet()
#18 /var/www/html/modules/nottinghamcollegeapi/src/services/GoogleBusinessProfile.php(175): verbb\consume\services\Service->fetchData()
#19 /var/www/html/modules/nottinghamcollegeapi/src/services/GoogleBusinessProfile.php(196): modules\nottinghamcollegeapi\services\GoogleBusinessProfile->_request()
#20 /var/www/html/modules/nottinghamcollegeapi/src/services/GoogleBusinessProfile.php(88): modules\nottinghamcollegeapi\services\GoogleBusinessProfile->_getGoogleBusinessProfileLocation()
#21 /var/www/html/modules/nottinghamcollegeapi/src/controllers/GoogleBusinessProfileController.php(71): modules\nottinghamcollegeapi\services\GoogleBusinessProfile->getFletchersRestaurantClosureDates()
#22 [internal function]: modules\nottinghamcollegeapi\controllers\GoogleBusinessProfileController->actionGetFletchersRestaurantClosedDates()
#23 /var/www/html/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array()
#24 /var/www/html/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams()
#25 /var/www/html/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction()
#26 /var/www/html/vendor/craftcms/cms/src/web/Application.php(350): yii\base\Module->runAction()
#27 /var/www/html/vendor/yiisoft/yii2/web/Application.php(103): craft\web\Application->runAction()
#28 /var/www/html/vendor/craftcms/cms/src/web/Application.php(318): yii\web\Application->handleRequest()
#29 /var/www/html/vendor/yiisoft/yii2/base/Application.php(384): craft\web\Application->handleRequest()
#30 /var/www/html/web/index.php(12): yii\base\Application->run()
#31 {main}”.

If I disconnect and reconnect it works for one hour, before the same issue.

Request had insufficient authentication scopes. is an interesting one. I'm not sure why the request is invalid evne though the initial request token works fine.

[engram-design]

Thanks for confirming. What sort of template calls and requests are you making so I can replicate?

[jamesmacwhite]

@engram-design Thanks Josh.

Example one:

https://mybusiness.googleapis.com/v4/accounts/$accountId/locations/$locationId/reviews

Get the reviews of an owned business listing accountId and locationId need to match an owned business listing account the location ID. I can't provide ours as you need permissions under the account context for it to work. Hoping Verbb has one somewhere!?

The requests work fine initially, then start throwing a 403 after the token expires. The 403 error is interesting, because it doesn't seem like the typical invalid token error, which I'd assume would be 401.

[engram-design]

Hmm, so I can confirm that things have been working here after the token has expired (which triggered a new token to refresh). It could be due to a service account, so I'll have to test with that. But yes, a 401 response is really the only one we issue a refresh token on.

[jamesmacwhite]

Interesting. I'm wondering it could be specific to our domain as we are under a Google Workspace environment. The OAuth is performed under a managed domain account. I'm wonder if moving off to non workspace account with permissions may be better. It has worked fine for a long time, but maybe something environment related is the reason.

I don't think we are using a service account for Google Business Profile, it has to be a Google account to authenticate.

[jamesmacwhite]

Thanks for testing though. I think it's environment specific as in our domain.

[engram-design]

Oh, my mistake for some reason I thought you were using a service account! I was actually using an account under a workspace as well. Still looking into it!

[jamesmacwhite]

I'm at a loss, I just can't understand why the Google provider suddenly doesn't work. I've tried various things on my end, new OAuth app, set to production, using a non-workspace account, adding specific OAuth scopes on the app being authenticated with, but no dice either.

The only thing to go on is the fact the response is a 403 error and doesn't seem like it is a typical OAuth auth issue.

I might have to contact Google... Wish me luck.

[jamesmacwhite]

I was able to dump the full error as the logs get truncated. Something doesn't like the https://www.googleapis.com/auth/business.manage scope, but that's literally the only scope required by the Google Business Profile documentation.

I've made a support ticket to the Google Business Profile API team, if it working for you, then I can't see it being anything with Consume, it was a long shot anyway given the Google Provider hasn't changed. I'm going to suggest it is something specific to our Google Cloud Console.

array(1) {
    [
        "error"
    ]=>
  array(4) {
        [
            "code"
        ]=>
    int(403)
    [
            "message"
        ]=>
    string(47) "Request had insufficient authentication scopes."
    [
            "status"
        ]=>
    string(17) "PERMISSION_DENIED"
    [
            "details"
        ]=>
    array(1) {
            [
                0
            ]=>
      array(4) {
                [
                    "@type"
                ]=>
        string(40) "type.googleapis.com/google.rpc.ErrorInfo"
        [
                    "reason"
                ]=>
        string(31) "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
        [
                    "domain"
                ]=>
        string(14) "googleapis.com"
        [
                    "metadata"
                ]=>
        array(2) {
                    [
                        "service"
                    ]=>
          string(44) "mybusinessbusinessinformation.googleapis.com"
          [
                        "method"
                    ]=>
          string(62) "google.mybusiness.businessinformation.v1.Locations.GetLocation"
                }
            }
        }
    }
}
array(1) {
    [
        "error"
    ]=>
  array(4) {
        [
            "code"
        ]=>
    int(403)
    [
            "message"
        ]=>
    string(47) "Request had insufficient authentication scopes."
    [
            "status"
        ]=>
    string(17) "PERMISSION_DENIED"
    [
            "details"
        ]=>
    array(1) {
            [
                0
            ]=>
      array(4) {
                [
                    "@type"
                ]=>
        string(40) "type.googleapis.com/google.rpc.ErrorInfo"
        [
                    "reason"
                ]=>
        string(31) "ACCESS_TOKEN_SCOPE_INSUFFICIENT"
        [
                    "domain"
                ]=>
        string(14) "googleapis.com"
        [
                    "metadata"
                ]=>
        array(2) {
                    [
                        "service"
                    ]=>
          string(44) "mybusinessbusinessinformation.googleapis.com"
          [
                        "method"
                    ]=>
          string(62) "google.mybusiness.businessinformation.v1.Locations.GetLocation"
                }
            }
        }
    }
}

[jamesmacwhite]

@engram-design If you've been able to get access to the Google Business Profile APIs and have an OAuth client setup, can I be really cheeky and ask to test the OAuth through it and see if I get the same 403 error when trying to refresh an obtained token.

I suspect the issue is with our Google Cloud Console, but testing the Google Business Profile APIs outside of our own is tricky, because you have to go through the Google Business Profile API request form to get access and outside of my employer, I have no real way to qualify for being an "organisation".

I want to use another Client ID and secret outside of our own cloud console, to rule that in or out.

[engram-design]

I was actually trying to get a call setup using that endpoint, but couldn't figure out how to get the accountId and locationId.

Maybe we chat through [email protected] and we can throw some details around?

[jamesmacwhite]

I might have found the issue, but I don't understand why it is happening at all. See what you think. This quite the wild ride, if anyone hits this in the future, I hope this helps you, because I've smacked my head against the wall for days on this.

Using the https://www.googleapis.com/oauth2/v1/tokeninfo?access_token=xxxx endpoint, I got the info of an access token that I know works. I generated a brand new fresh token to be fair on the testing, cleared my OAuth connection entirely, started fresh.

This token is from a general Google account that has access to our Google Business Profile groups, I don't think it matters, but in general troubleshooting steps I wanted to try and bypass our Google Workspace accounts just to eliminate some domain specific stuff happening and technically all our Google Workspace accounts have 2FA enabled through SSO which I believe started this entire mess of requiring a OAuth reconnect, so maybe long term better to do this anyway. We can't use service accounts through Google Business Profile so a Google account is always needed and OAuth is the only supported method.

The access token that works with some details masked:

{
    "issued_to": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "audience": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "user_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "scope": "https://www.googleapis.com/auth/business.manage https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid",
    "expires_in": 3550,
    "email": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "verified_email": true,
    "access_type": "offline"
}

That all looks good, we have our offline type which means we will get a refresh token to handle automatic refreshing and indeed things work great until we hit the expires_in value which starts the chain of events.

Now when the next request is made when the expires_value is no longer in the future, this triggers the Auth module (through Consume), given it is the owner of the token to refresh token. All good, that's what Consume is designed to do to make OAuth requests easier without worrying about tokens and indeed does its job here, we can see the database row has been updated by the dateUpdated column and a new access token provided, so nothing wrong with this, but as we know any requests to Google Business Profile now throw this strange 403 error. It's not a 401, so the token isn't "invalid" but 403 generally is usually permissions related in this context so there's something not right.

What seems to be happening and only when viewing the details of that new access token from the refresh process, a crucial change has occurred in the scope value. Notice, that the https://www.googleapis.com/auth/business.manage scope has been removed entirely in the new token that was obtained from the refresh:

{
    "issued_to": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "audience": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "user_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "scope": "https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid",
    "expires_in": 3570,
    "email": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    "verified_email": true,
    "access_type": "offline"
}

Bingo. This explains the 403 error and the insufficient scope error, given the scope needed to work has been completely removed from the token, but as to why it has been removed I have no idea and is still bizarre. It does however at least provide some detail on the 403 error. All the information I viewed about this was just saying "oh you must be missing a scope". Yeah turns out I am, a rather important one, the single scope needed to call Google Business Profile APIs that was originally requested in the first place! It doesn't really make sense because this is set on the provider and was present on the original token and shouldn't be removed by anything, but it appears that's what is happening and why the whole world comes crashing down. Arguably the error has been there all along and was pretty clear when you now see the token issue, just the reason for it doesn't really add up.

So the problem has been found, the refreshed access token has lost the original scope needed, but why has it been removed and what has caused it to be removed? This remains the mystery.

I'm still unsure of anything in Consume being wrong, after all it is just uses the League Google provider which as far as I know hasn't changed for a while, we have other OAuth providers configured that have scopes added and these work fine, granted they aren't Google providers but it rules out something potentially wrong with that field or not being saved in the model. Equally it clearly is sent on the initial OAuth token request with no problems.

I have an API support ticket with the Google Business Profile team, thus far I'm mostly getting responses which basically translate to "skill issue", so thanks Google, but OAuth shouldn't be this hard... I really don't know what to make of it. Who's heard of OAuth refresh tokens just dropping scopes randomly?! I'd get it if say for example the access_type wasn't set to offline or something, as then we just wouldn't get a refresh token at all, but a refresh token that just drops the one scope requested, given all the others are defaults in the provider anyway. WHAAAAAAAAAAAAAT.

Either there is a edge case bug somewhere or the Google OAuth process specifically for Google Business Profile API calls has decided to break recently...

I do now feel a bit better about understanding the problem however, but I still need to find a solution somehow!

I'll pass on my findings to the support ticket I have with Google, maybe just before Christmas I'll get a miracle!

[engram-design]

Thanks for the detailed write-up. I didn't actually realise a new token was being generated, but as you outlined it was, which makes what was going on all the more stranger.

But, that missing scope is certainly not correct, I'm not sure how I missed that! I'd say that's almost certainly something on my end, so I'll investigate that! I note that's the response from Google with those details, but maybe I'm doing something wrong.

Just to be sure, you're setting this client up through the UI, correct? Not adding any scopes through a config file, or through templates?

[jamesmacwhite]

Thanks! Yes can confirm I have configured the Google Provider through the CP, not through a config file or templates. We have another OAuth provider connecting to Microsoft SharePoint and that's a custom provider class and that works fine.

The thing is, the Google provider has worked fine with that scope for literally months and only recently has it started happening, however from checking the git commit logs, I cannot see any major change which would have done anything. While it might not be as straightforward to compare given technically Consume is leveraging the Auth module for many OAuth things, I still can't see anything in the Auth module either.

Likewise the timeline of when things started not working was within the past few weeks, this does not align to any release. So that's what bothers me. There is no logical reason for this to have just started breaking.

I'm going to have a hard time probably telling Google its something on their end, but I honestly can't see what has changed on the Craft CMS side. There's been Craft CMS version updates, that's all that it could be really.

[engram-design]

Just checking in on my refresh tokens, and my scopes are retained. In my example client, they are:

https://www.googleapis.com/auth/analytics https://www.googleapis.com/auth/business.manage https://www.googleapis.com/auth/analytics.edit https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/userinfo.profile openid

I can't really find any resources about how the scopes are missing from the refresh token though. We don't send them back again for the accessToken request, but I don't believe you typically do.

[jamesmacwhite]

Interesting. I don't think the issue is with Consume doing anything wrong. It actually makes sense for it to not be it, given no major changes being made.

I think there's something specific going on in our Google Cloud Console or on the Google end, I passed the info on to Google API support. At least I have something tangible as to why it doesn't work. It will be now a case of finding out what removes the scope in the first place.

I am trying to get access to the Google Business Profile APIs under another Google Cloud Console project so I can confirm the theory. Of course these API are by request only...

[jamesmacwhite]

I'm at a loss of why it doesn't work. I can only hope I can get Google Business Profile access under another Google Cloud Project to test if a different Client ID and secret from another project that's not connected to our domain workspace does anything. It is all I can think of.

[jamesmacwhite]

The only thing I can think of is refreshing the token before the expiry, that potentially might work to stop the scope being lost. Is there a way to do that through the Consume/Auth services? I could run it as a console command or cron, to ensure it runs frequently to essentially keep the token alive before it expires.

It isn't the proper fix, but until I can get somewhere with Google support, it seems like a workable option for now.

Also, Happy Christmas for Australia being 11 hours ahead!

[engram-design]

We don't have a console command for that, but I like the idea of one!

Added for the next release. To get this early, run composer require verbb/consume:"dev-craft-5 as 2.0.1".

./craft consume/tokens/refresh --handle=google

[jamesmacwhite]

Happy Christmas! That's great, thank you for that!

[jamesmacwhite]

Well, it helps show the issue we are having easily as soon as the token gets refreshed I lose the scope needed, but at least it can be replicated quicker than 1 hour for Google support!

I can see it being useful for other contexts though, I think we are hitting a rather unique issue. If you are not losing scopes on refreshing, I'm now near 100% confident, something with our Google Cloud project is in play, but it is not as if it can be configured wrong when it works for the the first token at least.

I'm hoping Google give me access to the APIs on another project outside of our domain, that will confirm it all.

[engram-design]

Good to know it's the process of refreshing that's triggering this. Now, one thing I must admin - I've just realised I've been testing with the wrong API (analytics), and that Google Business Profile is an invite-only API. I'll request for access, but I feel like this whole API might be the deciding difference.

[jamesmacwhite]

Unfortunately, Google Business Profile APIs are indeed request only, so debugging them is difficult. Under the same Google Cloud Console, you are also only allowed one project to be enabled for them as well, just to make it more fun.

I've got some active channels open with a few different Google support areas. I've provided a video recently showing the problem to them. Basically, comparing the two tokens in Postman, showing the use of a refresh token (Generated by the Google OAuth API) is just nuking the original scope requested. I cannot see how anything on the client or Consume end is doing that, when that scope is present to start with. This also didn't start happening until about 2 weeks back. If it is on our end, I feel it is specific to our Google Workspace/Google Cloud console at this point. However, it is not as if we are actively removing business.manage scopes from tokens, but if there is a configuration doing that somewhere in Google Cloud Console/workspace then that would explain it.

[jamesmacwhite]

Just to test, I put in another unrelated scope https://www.googleapis.com/auth/analytics.readonly in addition to the Google Business Profile one. As soon as it got refreshed both were missing from the token entirely, so I'm not sure its API specific at this point.

If you aren't seeing that behaviour, then I'm more convinced something specific to our Google Workspace is doing something.

[jamesmacwhite]

So another find, which just adds to the mystery but I've technically got a workaround so I can at least fix the problem for now.

Using the Consume refresh token method, I lose the scope when the refresh is performed as we know, if I send a refresh_token request directly to https://oauth2.googleapis.com/token and do something like:

$response = Craft::createGuzzleClient()
    ->post('https://oauth2.googleapis.com/token', [
        'form_params' => [
            'client_id' => App::parseEnv($client->clientId),
            'client_secret' => App::parseEnv($client->clientSecret),
            'grant_type' => 'refresh_token',
            'refresh_token' => $token->refreshToken
        ]
    ]);

The scope is maintained from the new access token and I can use this with no issues.

So while I shouldn't need to do it. I can insert the new access token and token values into the DB row and update the expires value and just have this run regularly enough, basically preventing the Consume refreshToken method from being triggered.

I have no idea why the refreshToken method through Consume is any different to calling the https://oauth2.googleapis.com/token through Guzzle, because I'm pretty sure https://oauth2.googleapis.com/token is part of the Google OAuth provider class, which ultimately gets called through a Guzzle HTTP client eventually?

[jamesmacwhite]

I think I have found the issue. It's way upstream: thephpleague/oauth2-client#1052.

The oauth2-client version 2.8.0 is specifically broken with refresh token handling. Downgrading back to 2.7.0 temporarily works for now, there is a PR pending by the looks of it. I don't think they'll need to be specific handling within Consume/Auth once a new release with PR merged is made.

At the moment though, Google APIs are quite broke by it, if you haven't got oauth2-client version 2.8.0 pulled in through Composer, you'd be OK. it is only when you have that version are refresh tokens broken.

[engram-design]

That’s a great find @jamesmacwhite !

[jamesmacwhite]

@engram-design A fun one to track down. You may temporarily want to force the oauth2-client package to 2.7.0 in Consume/Auth until it is fixed upstream, there's a few scenarios where it can break OAuth clients.