[BUG]: AzurePowerShell@5 is failing to install Az modules

[RickKra]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

AzurePowerShell

Task version

5.239.11

Issue Description

For these 3 builds:

• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9604550&view=logs&j=dba4eab8-d390-5c78-738f-db7f79514c75&t=c65817ea-71c0-54c4-1fb8-aa7f6abad7a3&s=571cbea6-1f1f-576e-ab76-5f97db473a7d
• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9604280&view=logs&j=dba4eab8-d390-5c78-738f-db7f79514c75&t=c65817ea-71c0-54c4-1fb8-aa7f6abad7a3&s=571cbea6-1f1f-576e-ab76-5f97db473a7d
• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9605511&view=logs&j=dba4eab8-d390-5c78-738f-db7f79514c75&t=c65817ea-71c0-54c4-1fb8-aa7f6abad7a3&s=571cbea6-1f1f-576e-ab76-5f97db473a7d

We got this, before our script even had a chance to run:

##[error]Package 'Az.Resources' failed to be installed because: End of Central Directory record could not be found.

For this build:

• https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9606096&view=logs&j=dba4eab8-d390-5c78-738f-db7f79514c75&t=c65817ea-71c0-54c4-1fb8-aa7f6abad7a3

We got this, also before our script could run:

##[error]No match was found for the specified search criteria and module name 'Az.Accounts'. Try Get-PSRepository to see all available registered module repositories.

This problem is intermittent.

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2019

Relevant log output

2024-05-20T19:33:33.7381204Z ##[section]Starting: Layout BuildArtifacts
2024-05-20T19:33:33.7385860Z ==============================================================================
2024-05-20T19:33:33.7385963Z Task         : Azure PowerShell
2024-05-20T19:33:33.7386023Z Description  : Run a PowerShell script within an Azure environment
2024-05-20T19:33:33.7386103Z Version      : 5.239.11
2024-05-20T19:33:33.7386150Z Author       : Microsoft Corporation
2024-05-20T19:33:33.7386207Z Help         : https://aka.ms/azurepowershelltroubleshooting
2024-05-20T19:33:33.7386274Z ==============================================================================
2024-05-20T19:33:34.4791927Z Generating script.
2024-05-20T19:33:34.5074276Z ========================== Starting Command Output ===========================
2024-05-20T19:33:34.5291831Z ##[command]"C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\a\_work\_temp\9d1676c0-d557-4875-81bb-3183c4e21c06.ps1'"
2024-05-20T19:33:35.4260872Z WARNING: String format failed.
2024-05-20T19:33:35.5239926Z ##[warning]Failed to Initialize Existing Az modules, fallback to installation
2024-05-20T19:33:35.5386423Z ##[command]Install-Module -Name Az.Accounts -Force -AllowClobber -ErrorAction Stop SkipPublisherCheck
2024-05-20T19:34:55.4908950Z ##[command]Import-Module -Name C:\Program Files\WindowsPowerShell\Modules\Az.Accounts\2.19.0\Az.Accounts.psd1 -Global -PassThru -Force
2024-05-20T19:34:56.6503461Z Imported module 'Az.Accounts', version: 2.19.0
2024-05-20T19:34:56.6564439Z ##[command]Update-AzConfig -DisplayBreakingChangeWarning False -AppliesTo Az
2024-05-20T19:34:56.8400777Z ##[command]Enable-AzureRmAlias -Scope Process
2024-05-20T19:34:57.5691157Z ##[command]Install-Module -Name Az.Resources -Force -AllowClobber -ErrorAction Stop SkipPublisherCheck
2024-05-20T19:38:55.9972009Z ##[error]Package 'Az.Resources' failed to be installed because: End of Central Directory record could not be found.
2024-05-20T19:38:56.0432033Z ##[error]Failed to Install Az modules
2024-05-20T19:38:56.1289362Z ##[error]PowerShell exited with code '1'.
2024-05-20T19:38:57.2853993Z ##[command]Disconnect-AzAccount -Scope CurrentUser -ErrorAction Stop
2024-05-20T19:38:57.3962503Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2024-05-20T19:38:57.4056340Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2024-05-20T19:38:57.5543188Z ##[section]Finishing: Layout BuildArtifacts

Full task logs with system.debug enabled

We don't have full task logs with system.debug enabled.

Repro steps

- task: AzurePowerShell@5
          displayName: Layout BuildArtifacts
          inputs:
            azureSubscription: 'VSEngCloudBuildAPI'
            workingDirectory: '$(Build.SourcesDirectory)'
            ScriptPath: 'src\Scripts\CB-BuildArtifacts.ps1'
            ScriptArguments: '-AccessToken $(System.AccessToken) -ReservedVersion $(ReservedVersion) -NugetPackageDirectory $(PackageDirectory) -ArtifactPublishDirectory $(ArtifactPublishDirectory) -DownloadDirectory $(DownloadDirectory) -VstsDropMetadataDownloadDirectory $(DropMetadataDirectory) -SourcesDirectory $(Build.SourcesDirectory) -BuildArtifactsJson $(BuildArtifactsJson)'
            azurePowerShellVersion: LatestVersion
          condition: succeeded()

[v-schhabra]

@RickKra Could you please install Az and Az.Accounts module externally and check if task works fine or not?

[RickKra]

@v-schhabra it's a complex pipeline building a production app, so I can't change it quickly and easily. Can you not fix the task?

[v-schhabra]

@RickKra I observed that now the task is working fine for you without any issues.
Could you please update if the issue is fixed or not?
https://devdiv.visualstudio.com/DevDiv/_build/results?buildId=9639822&view=logs&j=dba4eab8-d390-5c78-738f-db7f79514c75&t=c65817ea-71c0-54c4-1fb8-aa7f6abad7a3

[RickKra]

@v-schhabra it only happened 4 times for us. I created the bug in case it happened more.

Any way you can look into why it happened and either add retries or something to prevent future occurrences?

[dimpatel-MSFT]

@v-schhabra This issue has been happening for us. Please take a look at this - https://msdata.visualstudio.com/Database%20Systems/_build/results?buildId=127932808&view=logs&j=8515a029-22a6-5510-9e17-f5665af94c87&t=224a833e-b5fd-5143-4f10-7152cc46d42f

Can this be fixed please?

[v-schhabra]

@dimpatel-MSFT Could you pls enable debug logs for the pipeline?

[dimpatel-MSFT]

@v-schhabra Please refer to this for debug logs - https://msdata.visualstudio.com/Database%20Systems/_build/results?buildId=128100163&view=logs&j=8515a029-22a6-5510-9e17-f5665af94c87&t=224a833e-b5fd-5143-4f10-7152cc46d42f

[v-schhabra]

Hi @dimpatel-MSFT
Thanks for sharing the logs. We are investigating your issue and will share the updates soon.

[v-schhabra]

The above error may be caused by the PowerShell NuGet package provider not being installed in 1ES agents.
https://microsoft.github.io/PSRule/v2/troubleshooting/#windows-powershell-is-in-noninteractive-mode
As a workaround we suggested @dimpatel-MSFT to install the Nuget packages through Azure Powershell or Powershell task and this works fine for her.
For the permanent fix we are working with 1ES Hosted agent team and will post the latest updates here.

[v-schhabra]

Closing this out as issue is mitigated. Please raise the new request incase you face any new issues.