Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[REGRESSION]: AzureFileCopy@6 failing to load Az.Accounts v3.0.0 module #19910

Closed
5 of 7 tasks
MSylvia opened this issue May 24, 2024 · 18 comments
Closed
5 of 7 tasks

[REGRESSION]: AzureFileCopy@6 failing to load Az.Accounts v3.0.0 module #19910

MSylvia opened this issue May 24, 2024 · 18 comments
Labels
Area: Release Area:RM RM task team bug regression This used to work, but a change in the service/tasks broke it. triage

Comments

@MSylvia
Copy link
Member

MSylvia commented May 24, 2024

New issue checklist

Task name

AzureFileCopy

Breaking task version

6.240.1

Last working task version

6.239.x

Regression Description

We have a pipeline that is using AzureFileCopy@6 with a managed identity. It has been in place for about 2 weeks and has been running without issue. On Tuesday the task started failing with the only visible change in the logs being the Az.Accounts module updating from v2.9.1 to v3.0.0.

When it fails, we get this error:

Failed to perform Auto-login: PSContextCredentialGet-AzAccessToken: -REDACTED- 'Get-AzAccessToken' command was found in the module 'Az.Accounts', but the module could not be loaded. For more information, run 'Import-Module Az.Accounts'.

In failed runs the listed Az.Accounts module version is always v3.0.0.

Imported module 'Az.Accounts', version: 3.0.0

Runs that executed earlier which passed have a version of v2.9.1.

Imported module 'Az.Accounts', version: 2.9.1

Cross contamination of jobs is not possible because a fresh machine with the exact same configuration is used for each run and that configuration has not changed.

I believe it may have been caused by this change from AzureRM to Az: f5b22d9

Example runs:

Pass: v2.9.1
https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9600629&view=logs&j=4736f78b-748e-51a8-c3dc-ea7706a06c3d&t=f124d820-a316-55db-635e-ba56d272505a
Fail: v3.0.0
https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9613411&view=logs&j=4736f78b-748e-51a8-c3dc-ea7706a06c3d&t=f124d820-a316-55db-635e-ba56d272505a

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows 11

Relevant log output

Starting: Upload to Storage
==============================================================================
Task         : Azure file copy
Description  : Copy files to Azure Blob Storage or virtual machines
Version      : 6.240.1
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
==============================================================================
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\3.0.0\Az.Accounts.psd1 -Global -PassThru -Force
##[warning]Both Az and AzureRM modules were detected on this machine. Az and AzureRM modules cannot be imported in the same session or used in the same script or runbook. If you are running PowerShell in an environment you control you can use the 'Uninstall-AzureRm' cmdlet to remove all AzureRm modules from your machine. If you are running in Azure Automation, take care that none of your runbooks import both Az and AzureRM modules. More information can be found here: https://aka.ms/azps-migration-guide
Imported module 'Az.Accounts', version: 3.0.0
Update-AzConfig -DisplayBreakingChangeWarning False -AppliesTo Az
Enable-AzureRmAlias -Scope Process
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Resources\1.13.0\Az.Resources.psd1 -Global -PassThru -Force
Imported module 'Az.Resources', version: 1.13.0
Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Storage\1.14.0\Az.Storage.psd1 -Global -PassThru -Force
Imported module 'Az.Storage', version: 1.14.0
Uninstall-AzureRm
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Connect-AzAccount 

......

Failed to perform Auto-login: PSContextCredentialGet-AzAccessToken: -REDACTED- 'Get-AzAccessToken' command was found in the module 'Az.Accounts', but the module could not be loaded. For more information, run 'Import-Module Az.Accounts'.

Full task logs with system.debug enabled

We don't have full task logs with system.debug enabled.

Repro steps

- task: AzureFileCopy@6
  displayName: 'Azure File Copy'
  inputs:
    SourcePath: '${{ parameters.BuildPackages }}'
    azureSubscription: ${{ parameters.AzureSubscription }}
    Destination: AzureBlob
    storage: ${{ parameters.AzureStorage }}
    ContainerName: ${{ parameters.AzureContainerName }}
    BlobPrefix: ${{ parameters.AzureUploadLocation }}
  condition: ${{ parameters.condition }}
@MSylvia MSylvia added bug regression This used to work, but a change in the service/tasks broke it. labels May 24, 2024
@v-schhabra v-schhabra added the Area:RM RM task team label May 27, 2024
@v-snalawade
Copy link
Contributor

@MSylvia - Could you please run the pipeline with debug log enabled and share the logs.

@MSylvia
Copy link
Member Author

MSylvia commented May 29, 2024

@v-snalawade
To test if the warning Both Az and AzureRM modules were detected on this machine was the cause, I ran a different pipeline on two modified images and got the same error.

With warning (Both Az and AzureRM on machine):

Without warning (Only Az):

Both runs in this pipeline have debug logs enabled.

@v-snalawade
Copy link
Contributor

@MSylvia - I dont see debug logs enabled, please add variable system.debug to true and run pipeline again.

@MSylvia
Copy link
Member Author

MSylvia commented May 30, 2024 •
edited
Loading

@v-snalawade Here is a new run with debug logs enabled.

With warning (Both Az and AzureRM on machine):

Without warning (Only Az):

@v-snalawade
Copy link
Contributor

@MSylvia -
I went through logs and these are not debug logs.
please add variable system.debug to true in pipeline variables and run pipeline again, you will get the detailed logs.

@MSylvia
Copy link
Member Author

MSylvia commented Jun 3, 2024

@v-snalawade Here is a new run with debug logs actually enabled this time. I mistakenly thought that enabling diagnostics also set debug to true before.

With warning (Both Az and AzureRM on machine):

Without warning (Only Az):

@jonathanjyi
Copy link

jonathanjyi commented Jun 4, 2024 •
edited
Loading

I'm running into the same issue:
Breaking task version
6.240.1
6.239.x

It seems like both 240 and 239 are broken when using Az.Accounts v3.0.0

@jonathanjyi
Copy link

Debug Logs here:
https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=1

@MSylvia
Copy link
Member Author

MSylvia commented Jun 4, 2024

@DmitriiBobreshev I see you made some changes recently to the Azure File Copy tasks. Do you know what might cause the issues we are seeing?

@v-snalawade
Copy link
Contributor

Debug Logs here: https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28468351&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=1

Thanks for the logs, will check and keep you posted on updates

@jonathanjyi
Copy link

jonathanjyi commented Jun 5, 2024 via email

@DmitriiBobreshev
Copy link
Contributor

Hi @MSylvia, @jonathanjyi, Most probably the changes not causing the issue as the changes FF for AzureRM migration were ON for a while for both organizations.
I see that both pools are using the same image and suspect that there is some inconsistency between PowerShell where the modules are enabling and pwsh which is used for Get-AzAccessToken CmdLet inside AzCopy.exe.
Moreover, the other CmdLets as Connect-AzAccount or Set-AzContext(both provided by Az.Accounts) succeeded, which tells us that the module was loaded correctly for PowerShell session but probably wasn't loaded for pwsh.
Is it possible to try another a pool with different image to check will it works for you?

@jonathanjyi
Copy link

jonathanjyi commented Jun 5, 2024 via email

@MSylvia
Copy link
Member Author

MSylvia commented Jun 6, 2024

@DmitriiBobreshev While switching pools / images worked for @jonathanjyi there still may be a problem because that pool is loading the older version of Az.Accounts.

Imported module 'Az.Accounts', version: 2.19.0

https://dev.azure.com/mseng/VS%20IDE%20-%20Telemetry/_build/results?buildId=28477100&view=logs&j=a5781d42-2f3f-596b-1540-70f475b4668a&t=62431333-4809-58f9-64bf-05d4a003a852&l=92

If/when that pool updates it could still be a problem.

@DevTestUser123
Copy link

Hi all any update on this? getting same issue

@tomaeda tomaeda mentioned this issue Jun 20, 2024
Closed
7 tasks
@MSylvia
Copy link
Member Author

MSylvia commented Jul 5, 2024

Since pinning the task version also fails, it seems like it could be related to this issue: #16505

@v-schhabra v-schhabra mentioned this issue Jul 8, 2024
Merged
2 tasks
@v-schhabra
Copy link
Contributor

#20079 This is the fix created for the above issue. Will update here once the fixes are deployed to all the rings.

@v-schhabra
Copy link
Contributor

Fix is deployed to all the rings with task version 6.242.2. Could you all please check and update if you are still facing the issues?
@MSylvia @jonathanjyi @DevTestUser123

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Area: Release Area:RM RM task team bug regression This used to work, but a change in the service/tasks broke it. triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@MSylvia @v-snalawade @v-schhabra @jonathanjyi @DmitriiBobreshev @DevTestUser123