[BUG]: Exit 57005 StackOverflowException using AzurePowershell@5 #20156
Comments
|
I'm attaching the |
|
This is happening 80% of the time. For the 20% of the time, it does work. Please help. Thanks muchly! |
|
We have decided to pin to an old version of AzurePowerShell@5 until this is fixed or figured out. That works great. |
|
Hi @trevors20 |
|
It'd heavily impacted our VS builds across pipelines before we pinned to the old version as a workaround to unblock folks. You can get all the failed build IDs from below query if comment out the last line (| summarize) if that helps: Go to Log Analytics and run query let _start = datetime('2024-07-12T00:00:00Z');
|
|
Hi. Same. We've started experiencing these stack overflows today, with that AzurePowershell@5 task. Testing now version 4... |
|
Thanks @fernando-lopezblanco-lrn . If it helps, we pinned to version 5.241.5 and that has stabilized our builds. -task: [email protected] |
|
Thanks @trevors20 for very quick response. Will try that version later on. Same issue with AzurePowershell@4 (shown as Version : 4.242.0 in the pipeline logs). Testing now @5.241.5 |
|
@5.241.5 is looking good, but other pipelines with @4 are starting to fail as well |
|
@fernando-lopezblanco-lrn , I'm not sure which version to pin to for the AzurePowershell@4. But, you should be able to find that by looking at your older builds that were working before you started using the 4.242.0 version of the task. At the top of the AzurePowershell@4 step in the build, you should see a header at the top of the logs. Something like this: Task : Azure PowerShell If you work backwards and find one of your older builds with a previous version than the 4.242.0 version then try pinning the task to the previous version that you find. To give a rough timeframe, I know that the AzurePowershell@5 task was updated a couple of weeks ago which started the instability here. So, you may need to look back a couple of weeks or so. :( |
|
@fernando-lopezblanco-lrn , actually, we should be able to see the versions in the source code history. The version is specified in the task.json for the task. In this case for AzurePowershell@4, that would be here: https://github.com/microsoft/azure-pipelines-tasks/commits/master/Tasks/AzurePowerShellV4/task.json Looking at the history, the change that was checked in a couple of weeks ago, if you look at the previous change it looks like you may want to pin to 4.241.2. Try it and see? |
|
@trevors20 yeah, the version 4.242.0 I gave you above, I took it from the header of the logs as you later mentioned. |
|
Curious if anyone has seen this issue recently. We took the pinned version workaround, and I'm looking to unwind it if it's fixed. |
|
Was able to get this to reproduce on 5.243.7 |
|
This bug seems it may be mitigated also by switching to pwsh mode instead of legacy powershell. Having good success with this if you wanted to unpin the version |
|
Also have this issue. Waiting for advices. |
|
Hi @trevors20, In order to proceed with investigation, could please share following details?
|
|
Hi @trevors20 |
|
We're seeing this too, earliest seen build affected is 5.245.1 and still seeing it in 5.245.3 |
|
@onetocny , @v-schhabra
|
|
We have identified the root cause in If there is a need to use latest AzurePowerShell tasks immediately there is a confirmed workaround. You can prepend following task before the failing AzurePowerShell tasks: - task: PowerShell@2
inputs:
targetType: 'inline'
script: |
Install-Module Az.Accounts -Force -AllowClobber -ErrorAction Stop -SkipPublisherCheck
Import-Module Az.Accounts -Global -PassThru -Force
Update-AzConfig -CheckForUpgrade $false |
|
I'm intermittently hitting stack overflow with version Unfortunately, I don't have the call stack from my failed run so I am extrapolating here. If we look at the callstack from the linked Az Powershell issue , the stackoverflow is around assembly resolution. The assembly resolver that's causing the stack overflow is a PowerShell ScriptBlock: Digging further, it gets into the PowerShell internal where it seems to be triggering an error: Looking at what the GetContextFromTLS does: internal ExecutionContext GetContextFromTLS()
{
ExecutionContext executionContextFromTLS = LocalPipeline.GetExecutionContextFromTLS();
if (executionContextFromTLS == null)
{
string original = ToString();
original = ErrorCategoryInfo.Ellipsize(CultureInfo.CurrentUICulture, original);
PSInvalidOperationException ex = PSTraceSource.NewInvalidOperationException(ParserStrings.ScriptBlockDelegateInvokedFromWrongThread, original);
ex.SetErrorId("ScriptBlockDelegateInvokedFromWrongThread");
throw ex;
}
return executionContextFromTLS;
}It is trying to emit a Searching azure-pipeline-tasks for cases where an assembly resolver is registered with powershell script we find two instances: It turns out, that in Azure PowerShell the VstsAzureHelpers module is loaded. So, to truly fix all the cases where this can happen, VstsAzureHelpers_ would need to be updated to use a c# assembly resolver instead of a powershell script block in VstsAzureHelpers cc @starkmsu who introduced this change Call stack for reference |
New issue checklist
Task name
AzurePowerShell
Task version
5.242.1
Issue Description
We are seeing a lot of Exit 57005 build breaks when using AzurePowerShell@5.
It looks like from the logs there is a StackOverflowException exception that is happening.
Was wondering if you might be able to help debug and fix this and/or help diagnose why this might be happening and provide a workaround or anything like that. Much appreciated.
Environment type (Please select at least one enviroment where you face this issue)
Azure DevOps Server type
dev.azure.com (formerly visualstudio.com)
Azure DevOps Server Version (if applicable)
No response
Operation system
Windows 2022
Relevant log output
Full task logs with system.debug enabled
Repro steps
The text was updated successfully, but these errors were encountered: