[BUG]: Azure File transform task. EPERM: operation not permitted, stat 'C:\agent\_work\r1\a\build_api\Jobs\App_Data\jobs\continuous\protocols\bin\2d35ec66-b8b4-4f9b-8a50-753889deb977.tmp'

[yuriykuts]

New issue checklist

• I searched for existing GitHub issues
• I read pipeline troubleshooting guide
• I checked how to collect logs

Task name

File transform

Task version

1.235.1

Issue Description

We are using the File Transform task to substitute variables in a JSON file within the release pipeline, and we are encountering an issue with the error:
##[error]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
This error is reproducible on different agents.

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows Server 2019 Standart

Relevant log output

2024-05-30T13:40:37.9775830Z ##[error]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent\_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9785834Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent\_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9788081Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent\_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9987040Z ##[section]Finishing: Jobs configuration transform

Full task logs with system.debug enabled

2024-05-30T13:40:35.2226188Z ##[debug]Evaluating condition for step: 'Jobs configuration transform'
2024-05-30T13:40:35.2228803Z ##[debug]Evaluating: succeeded()
2024-05-30T13:40:35.2229335Z ##[debug]Evaluating succeeded:
2024-05-30T13:40:35.2230240Z ##[debug]=> True
2024-05-30T13:40:35.2231883Z ##[debug]Result: True
2024-05-30T13:40:35.2232957Z ##[section]Starting: Jobs configuration transform
2024-05-30T13:40:35.3127120Z ==============================================================================
2024-05-30T13:40:35.3127540Z Task : File transform
2024-05-30T13:40:35.3127712Z Description : Replace tokens with variable values in XML or JSON configuration files
2024-05-30T13:40:35.3128012Z Version : 1.235.1
2024-05-30T13:40:35.3128303Z Author : Microsoft Corporation
2024-05-30T13:40:35.3128491Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/file-transform
2024-05-30T13:40:35.3128783Z ==============================================================================
2024-05-30T13:40:37.9664033Z ##[debug] C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin (directory)
2024-05-30T13:40:37.9727460Z ##[debug]task result: Failed
2024-05-30T13:40:37.9775830Z ##[error]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9785834Z ##[debug]Processed: ##vso[task.issue type=error;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9788081Z ##[debug]Processed: ##vso[task.complete result=Failed;]Error: Failed find: EPERM: operation not permitted, stat 'C:\agent_work\r56\a\build_api\Jobs\App_Data\jobs\continuous\innovations\bin\5d7bf686-1f94-4ce8-8956-21210b956f2d.tmp'
2024-05-30T13:40:37.9987040Z ##[section]Finishing: Jobs configuration transform

Repro steps

No response

[paulcurren]

The problem appears to arise in the download-pipeline-artifact task.

The artefacts are downloaded as chunks - the '.tmp' files. It seems that the download task exits before all the '.tmp' files are uncompressed into real files. The uncompressing carries on in the background after the download task has exited.

Adding a PowerShell script to add a 60 second delay after download will allow the next task to complete.

This is not a 'workaround'. The issue with the download task needs to be fixed.

[paulcurren]

Had to add a PowerShell script in a new pipeline stage to make sure the downloads are complete before allowing further deployment pipeline stages to complete:

while ($true)
{
 $tmpFileCount = (Get-ChildItem -Path MyWebApp -Filter *.tmp -Recurse).Count
 
 if ($tmpFileCount -eq 0)
 {
   break;
 }
 
 Write-Host "Waiting for $tmpFileCount .tmp files..."
 Start-Sleep -Seconds 5
}

Result:

2024-06-27T10:42:41.6678367Z Waiting for 4 .tmp files...
2024-06-27T10:42:46.6956047Z Waiting for 3 .tmp files...
2024-06-27T10:42:51.7231092Z Waiting for 3 .tmp files...
2024-06-27T10:42:56.7486519Z Waiting for 3 .tmp files...
2024-06-27T10:43:01.7796169Z Waiting for 1 .tmp files...
2024-06-27T10:43:06.7994872Z Waiting for 1 .tmp files...
2024-06-27T10:43:11.8147468Z Waiting for 1 .tmp files...
2024-06-27T10:43:16.8391393Z Waiting for 1 .tmp files...
2024-06-27T10:43:21.8675132Z Waiting for 1 .tmp files...
2024-06-27T10:43:26.8840815Z Waiting for 1 .tmp files...
2024-06-27T10:43:31.9054132Z Waiting for 1 .tmp files...
2024-06-27T10:43:36.9249268Z Waiting for 1 .tmp files...
2024-06-27T10:43:41.9504679Z Waiting for 1 .tmp files...

This is a kludge, not a workaround.

[CrommVardek]

The workaround provided by @paulcurren works, but is this bug will be investigated ? Why the uncompressing is done in the background after the download artifacts task is finished ? Why not make it part of the task, so it does not give a false signal that the files are ready for further processing and fully availables for the other tasks ?

[petertheautomator]

Any update on this?