Get-MgUserMailboxSettings returns Too many retries performed

[anthonyvandenbossche]

Describe the bug

For the most part of my organization's mailboxes, I can successfully retrieve mailboxsettings. However, for some of them I receive the following error (v1 and beta - cannot test on explorer): Get-MgBetaUserMailboxSetting_Get: Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: ServiceUnavailable.)

Invoke-MgGraphRequests returns the same error: Invoke-MgGraphRequest: Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: ServiceUnavailable.)

Expected behavior

This command should come back with a hashtable containing (among others), properties like: userPurpose, workingHours, language...

How to reproduce

Until this moment, I cannot reproduce it on another tenant and thus haven't found a common characteristic to a mailbox that would result in this behavior.

SDK Version

2.21.1

Latest version known to work for scenario above?

N/A

Known Workarounds

None at this moment.

Debug output

Click to expand log

```

DEBUG: [CmdletBeginProcessing]: - Get-MgBetaUserMailboxSetting begin processing with parameterSet 'Get'.
DEBUG: [Authentication]: - AuthType: 'AppOnly', TokenCredentialType: 'ClientCertificate', ContextScope: 'Process', AppName: '[REDACTED]'.
DEBUG: [Authentication]: - Scopes: [Group.Read.All, MailboxSettings.Read, Sites.Read.All, User.Read.All, Domain.Read.All, AuditLog.Read.All].
DEBUG: [CmdletException]: Received exception with message 'AggregateException - Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: GatewayTimeout.� �
T�M�� �@�̺��������L��Hk\4����ܾ/o��}��7(3�D��[������<dY�� �jM�O���֣k��)��ge�u�X����%ˊ�C��m�v���i4���j=�Ϫ$GT����}���ߪĄd����h�&LD����cr���}�? ��� ����� ) (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: GatewayTimeout.� � <l���h,�����T���#F���s���2'h���e���)��!��!;J(���E!5��@���,���h���4#Bv�c�)⑂���(&�m� ��� �ⵤ� ) (HTTP request failed with status code: GatewayTimeout.� �
T�K�� ���̚Ix콁�����FZ��ݍ�r���7�1��q�2W�~����Y���%]> ���y�$nP���ZjB�P铢h}4 �~<yYq��AY�B2��L��挩��d�9�d3ك�r�������RH��6$6�^��&áf�8x����� ��� :���� ) : at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendRetryAsync(HttpResponseMessage response, RetryHandlerOption retryOption, CancellationToken cancellationToken, ActivitySource activitySource) at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.RetryHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Kiota.Http.HttpClientLibrary.Middleware.CompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at Microsoft.Graph.PowerShell.Authentication.Handlers.AuthenticationHandler.SendAsync(HttpRequestMessage httpRequestMessage, CancellationToken cancellationToken) at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Microsoft.Graph.Beta.PowerShell.Users.UserGetMailboxSetting_Call(HttpRequestMessage request, Func3 on2Xx, Func3 onDefault, IEventListener eventListener, ISendAsync sender) at Microsoft.Graph.Beta.PowerShell.Users.UserGetMailboxSetting_Call(HttpRequestMessage request, Func3 on2Xx, Func3 onDefault, IEventListener eventListener, ISendAsync sender) at Microsoft.Graph.Beta.PowerShell.Users.UserGetMailboxSetting(String userId, String[] Select, String[] Expand, IDictionary headers, Func3 on2Xx, Func3 onDefault, IEventListener eventListener, ISendAsync sender) at Microsoft.Graph.Beta.PowerShell.Cmdlets.GetMgBetaUserMailboxSetting_Get.ProcessRecordAsync()' Get-MgBetaUserMailboxSetting_Get: Too many retries performed. More than 3 retries encountered while sending the request. (HTTP request failed with status code: GatewayTimeout.� � T�M�� �@�̺��������L��Hk\4����ܾ/o��}��7(3�D��[������<dY�� �jM�O���֣k��)��ge�u�X����%ˊ�C��m�v���i4���j=�Ϫ$GT����}���ߪĄd����h�&LD����cr���}�? ��� ����� ) (HTTP request failed with status code: ServiceUnavailable.) (HTTP request failed with status code: GatewayTimeout.� �
T�K��@@��5M�Nu>{o�(����q�� �ݠ+�ﳂ�:U�+���d��[�����<l���h,�����T���#F���s���2'h���e���)��!��!;J(���E!5��@���,���h���4#Bv�c�)⑂���(&�m� ��� �ⵤ� ) (HTTP request failed with status code: GatewayTimeout.� � T�K�� ���̚Ix콁�����FZ��ݍ�r���7�1��q�2W�~����Y���%]> ���y�$nP���ZjB�P铢h}4
�~<yYq��AY�B2��L��挩��d�9�d3ك�r�������RH��6$6�^��&áf�8x����� ��� :���� )
DEBUG: [CmdletEndProcessing]: - Get-MgBetaUserMailboxSetting end processing.

</details>


### Configuration

PSVersion                      7.4.5
PSEdition                      Core
GitCommitId                    7.4.5
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

### Other information

_No response_

[timayabi2020]

@anthonyvandenbossche just as you have stated, it's not easy to identify a common pattern for this issue. Please note that the SDK has an inbuilt retry handler whose default retry limit is set to 3. The retry is based on conditions outlined in https://github.com/microsoftgraph/msgraph-sdk-design/blob/master/middleware/RetryHandler.md. Also see the service specific limits described in Service-specific limits are described in https://docs.microsoft.com/en-us/graph/throttling.

Kindly also share the outcome of Get-MgRequestContext in the same session whenever you experience this issue.

[anthonyvandenbossche]

Hi, below the output from Get-MgRequestContext. We indeed notice that multiple retries are being done. This issue is currently very blocking because ShareGate uses this cmdlet to fetch information before mailbox migration. We have a premier call running that is not going anywhere as we speak..

ClientTimeout RetryDelay MaxRetry RetriesTimeLimit

---

00:05:00 3 3 00:00:00

I could give you a tenant ID (and a user account with the issue - we have 58 users with this issue) if that can help with troubleshooting the backend.

[anthonyvandenbossche]

Please also note that the backend error code is 504; GatewayTimeout. This is not a throttling issue, as I can reproduce the issue with a single command.