[BUG]: AzureFileCopy to VM fails #19685

AndrewBates666 · 2024-03-25T17:04:26Z

New issue checklist

Task name

AzureFilecopy

Task version

5.237.1

Issue Description

The AzureFileCopy command started failing last week after presumably being updated. The error being reported is:

2024-03-21T10:42:55.7596977Z -ExecutionPolicy Unrestricted -Command ".
2024-03-21T10:42:55.7597475Z 'C:\Users\adminuser\AppData\Local\Temp\0a0b6c85-26a3-4c18-803e-537ce07dce30.ps1'"
2024-03-21T10:42:55.7597735Z
2024-03-21T10:42:55.7739358Z VERBOSE: Using default AzCopy arguments for dowloading to VM
2024-03-21T10:42:55.7740270Z
2024-03-21T10:42:55.7787839Z VERBOSE: ##[command] & azcopy copy
2024-03-21T10:42:55.7788454Z
2024-03-21T10:42:56.4219789Z "https://XXXdevconfigstorage.blob.core.windows.net/e576feb6-e8bb-47a2-8806-55dc5fea1c13*****"
2024-03-21T10:42:56.4220051Z
2024-03-21T10:42:56.4220291Z "c:\temp" --recursive --log-level=INFO
2024-03-21T10:42:56.4220397Z
2024-03-21T10:42:56.4220445Z
2024-03-21T10:42:56.4220489Z
2024-03-21T10:42:56.4220763Z failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A
2024-03-21T10:42:56.4220980Z

Apparently the SAS token is being added to the URL but it is missing the ? separator.

We have implemented a workaround as proposed by another user using two tasks, one to upload to the blob and another PowerShell script to copy from the blob to the VM but this has started to fail this week.

The problem with the workaround is that it appears that the azcopy command is not being found on the Hosted Agent so the workaround no longer works for us. The error reported is:

2024-03-25T16:45:00.5157506Z azcopy : The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or
2024-03-25T16:45:00.5157839Z At line:1 char:1
2024-03-25T16:45:00.5158421Z + & 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoLogo ...
2024-03-25T16:45:00.5159704Z + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2024-03-25T16:45:00.5160428Z + CategoryInfo : NotSpecified: (azcopy : The te...cript file, or :String) [], RemoteException
2024-03-25T16:45:00.5160843Z + FullyQualifiedErrorId : NativeCommandError
2024-03-25T16:45:00.5161132Z
2024-03-25T16:45:00.5161307Z
2024-03-25T16:45:00.5264081Z operable program. Check the spelling of the name, or if a path was included, verify that the path
2024-03-25T16:45:00.5264566Z
2024-03-25T16:45:00.5344297Z is correct and try again.

Currently we can't release any of our software so development and testing teams are blocked.

Environment type (Please select at least one enviroment where you face this issue)

Self-Hosted
Microsoft Hosted
VMSS Pool
Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

Windows Server 2019/2022

Operation system

Windows Server 2019/2022

Relevant log output

Errors are inline above.

Full task logs with system.debug enabled

 [REPLACE THIS WITH YOUR INFORMATION]

Repro steps

An AzureFileCopy task to copy a file onto a VM was working correctly great before last week but started to fail. We noted that the build version of the task changed from 5.236.1 (worked okay) to 5.237.1 (fails).

gusAntonis · 2024-03-26T02:06:53Z

We are also facing the same issue with the AzureFileCopy task, trying to use workaround with azcopy to copy the builds to the VMs,
Those are also failing as well, all of the deployments have been blocked for the team.

v-mohithgc · 2024-03-26T08:19:50Z

Hi all, the task version is now reverted to 5.236.1. can you please rerun your pipeline and confirm back with the result. Thanks

AndrewBates666 · 2024-03-26T08:51:11Z

Seeing the same error with version 5.236.1.

gusAntonis · 2024-03-26T08:51:39Z

Re-ran the pipelines

Hi all, the task version is now reverted to 5.236.1. can you please rerun your pipeline and confirm back with the result. Thanks

Re-ran the deployment pipelines, are the Azure file copy task's revert didn't helped with the issue,
We have used the AzureFileCopy version 3,4 and 5 as well
The problem we are facing lies in the Azure PS 11.0.0+ . More specifically Az.Storage 6.0.0+. There was a breaking change in New-AzStorageContainerSASToken -- the return value is no longer prefixed with a question mark. The code of the tasks relies on that, though.

https://learn.microsoft.com/en-us/powershell/azure/release-notes-azureps#azstorage-600
https://learn.microsoft.com/en-us/powershell/azure/migrate-az-11.0.0#azstorage

The task log is not very helpful as it displays the source URI with five asterisks appended, while in reality, a slash, an asterisk, and the SAS token are appended.

The New-AzStorageContainerSASToken cmdlet is called in Generate-AzureStorageContainerSASToken defined in AzureUtilityAz1.0.ps1. That one is called from Copy-FilesToAzureVMsFromStorageContainer defined in Utility.ps1, which is called from AzureFileCopy.ps1 itself. There, it can be seen that Copy-FilesToAzureVMsFromStorageContainer gets the "fileCopyJobScript" scriptblock from AzureFileCopyRemoteJob.ps1. At the bottom, see the uses of containerSasToken: "$containerURL/*$containerSasToken"

AndrewBates666 · 2024-03-26T08:56:44Z

Transfer to the blob appeared to work okay but it's the azcopy to the VM that is raising the error. Confirmed that 5.236.1 was the last successful version run.

Log file for the failure is:

2024-03-26T08:48:40.4356045Z INFO: Logout succeeded.
2024-03-26T08:48:41.0590923Z INFO: AzCopy.exe: A newer version 10.23.0 is available to download
2024-03-26T08:48:41.0591071Z
2024-03-26T08:48:41.0600519Z INFO: AzCopy.exe: A newer version 10.23.0 is available to download
2024-03-26T08:48:41.0601575Z
2024-03-26T08:48:50.0901423Z PSSession created for Machines:'xxxbldevsol.uksouth.cloudapp.azure.com'
2024-03-26T08:48:50.4821322Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================
2024-03-26T08:48:55.4789167Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================
2024-03-26T08:48:55.4879442Z ##[command]& 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoLogo -NoProfile -NonInteractive
2024-03-26T08:48:55.4879760Z -ExecutionPolicy Unrestricted -Command ".
2024-03-26T08:48:55.4880058Z 'C:\Users\adminuser\AppData\Local\Temp\17c14967-f0bd-4737-a6b1-a9fa1354e0a1.ps1'"
2024-03-26T08:48:55.4930051Z Directory: C:
2024-03-26T08:48:55.4962444Z Mode LastWriteTime Length Name
2024-03-26T08:48:55.4973726Z ---- ------------- ------ ----
2024-03-26T08:48:55.4983831Z d----- 3/26/2024 8:48 AM ADO_AzCopyV10
2024-03-26T08:48:55.4994545Z VERBOSE: Preparing to expand...
2024-03-26T08:48:55.5004869Z VERBOSE: Created 'C:\ADO_AzCopyV10\AzCopy\AzCopy.exe'.
2024-03-26T08:49:00.4981637Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================
2024-03-26T08:49:00.5007325Z failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A

AndrewBates666 · 2024-03-26T09:00:33Z

Re-ran the pipelines

Hi all, the task version is now reverted to 5.236.1. can you please rerun your pipeline and confirm back with the result. Thanks

Re-ran the deployment pipelines, are the Azure file copy task's revert didn't helped with the issue, We have used the AzureFileCopy version 3,4 and 5 as well The problem we are facing lies in the Azure PS 11.0.0+ . More specifically Az.Storage 6.0.0+. There was a breaking change in New-AzStorageContainerSASToken -- the return value is no longer prefixed with a question mark. The code of the tasks relies on that, though.

https://learn.microsoft.com/en-us/powershell/azure/release-notes-azureps#azstorage-600 https://learn.microsoft.com/en-us/powershell/azure/migrate-az-11.0.0#azstorage

The task log is not very helpful as it displays the source URI with five asterisks appended, while in reality, a slash, an asterisk, and the SAS token are appended.

The New-AzStorageContainerSASToken cmdlet is called in Generate-AzureStorageContainerSASToken defined in AzureUtilityAz1.0.ps1. That one is called from Copy-FilesToAzureVMsFromStorageContainer defined in Utility.ps1, which is called from AzureFileCopy.ps1 itself. There, it can be seen that Copy-FilesToAzureVMsFromStorageContainer gets the "fileCopyJobScript" scriptblock from AzureFileCopyRemoteJob.ps1. At the bottom, see the uses of containerSasToken: "$containerURL/*$containerSasToken"

Great work Sherlock! ;o)

AndrewBates666 · 2024-03-26T09:02:43Z

Hi all, the task version is now reverted to 5.236.1. can you please rerun your pipeline and confirm back with the result. Thanks

See gusAntonis reply above. Adding the ? and rolling forward might be the simpler route as both seem dependent on that breaking change in the other module. Just change the line to or check to see if the $containerSasToken starts with a ? and if not add one:

"$containerURL/*?$containerSasToken"

gusAntonis · 2024-03-26T09:03:10Z

Transfer to the blob appeared to work okay but it's the azcopy to the VM that is raising the error. Confirmed that 5.236.1 was the last successful version run.

Log file for the failure is:

2024-03-26T08:48:40.4356045Z INFO: Logout succeeded. 2024-03-26T08:48:41.0590923Z INFO: AzCopy.exe: A newer version 10.23.0 is available to download 2024-03-26T08:48:41.0591071Z 2024-03-26T08:48:41.0600519Z INFO: AzCopy.exe: A newer version 10.23.0 is available to download 2024-03-26T08:48:41.0601575Z 2024-03-26T08:48:50.0901423Z PSSession created for Machines:'utcbldevsol.uksouth.cloudapp.azure.com' 2024-03-26T08:48:50.4821322Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================ 2024-03-26T08:48:55.4789167Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================ 2024-03-26T08:48:55.4879442Z ##[command]& 'C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe' -NoLogo -NoProfile -NonInteractive 2024-03-26T08:48:55.4879760Z -ExecutionPolicy Unrestricted -Command ". 2024-03-26T08:48:55.4880058Z 'C:\Users\adminuser\AppData\Local\Temp\17c14967-f0bd-4737-a6b1-a9fa1354e0a1.ps1'" 2024-03-26T08:48:55.4930051Z Directory: C: 2024-03-26T08:48:55.4962444Z Mode LastWriteTime Length Name 2024-03-26T08:48:55.4973726Z ---- ------------- ------ ---- 2024-03-26T08:48:55.4983831Z d----- 3/26/2024 8:48 AM ADO_AzCopyV10 2024-03-26T08:48:55.4994545Z VERBOSE: Preparing to expand... 2024-03-26T08:48:55.5004869Z VERBOSE: Created 'C:\ADO_AzCopyV10\AzCopy\AzCopy.exe'. 2024-03-26T08:49:00.4981637Z ================================================ utcbldevsol.uksouth.cloudapp.azure.com ================================================ 2024-03-26T08:49:00.5007325Z failed to parse user input due to error: cannot use wildcards in the path section of the URL except in trailing "/*". If you wish to use * in your URL, manually encode it to %2A

Can confirm the same behaviour for our pipeline as well,
Attaching logs for reference (removed the names for files and machines),

##[debug]Leaving Invoke-RemoteScript.
##[debug]Parallel file copy: Invoke-RemoteScript threw exception
##[debug]Microsoft.PowerShell.Commands.WriteErrorException: Non-Zero exit code: '1' for ComputerName: 'VIMBC001'
##[debug]Processed: ##vso[task.logissue type=error;code={"Task_Internal_Error":"CopyingToAzureVMFailed"};]
##[debug][Azure Call]Deleting container: 074e42-9fdd-a9289a670df7 in storage account: pstas001
##[debug]Removed container '07a402-9fdd-a9289670df7' successfully.
##[debug][Azure Call]Deleted container: 07a4e42-9fdd-a9289a670df7 in storage account: stosa001
##[debug]Completed Azure File Copy Task for Azure VMs Destination
##[debug]Leaving D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\AzureFileCopy.ps1.
##[debug]Trying to disconnect from Azure and clear context at process scope
##[debug]Cannot verify the Microsoft .NET Framework version 4.7.2 because it is not included in the list of permitted versions.
##[debug]Populating RepositorySourceLocation property for module Az.Accounts.
##[debug]Loading module from path 'C:\Modules\az_11.3.1\Az.Accounts\2.16.0\Az.Accounts.psm1'.
Disconnect-AzAccount -Scope Process -ErrorAction Stop
Clear-AzContext -Scope Process -ErrorAction Stop
##[debug]Caught exception from task script.
##[debug]Error record:
##[debug]Set-TaskResult : Non-Zero exit code: '1' for ComputerName: 'IMBC001'
##[debug]At D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\ps_modules\RemoteDeployer\RemoteDeployer.psm1:134 char:9
##[debug]+         Set-TaskResult -jobResults $jobResults -machinesCount $target ...
##[debug]+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
##[debug]    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
##[debug]    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Set-TaskResult
##[debug] 
##[debug]Script stack trace:
##[debug]at Set-TaskResult, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\ps_modules\RemoteDeployer\RemoteDeployer.Utility.ps1: line 138
##[debug]at Invoke-RemoteScript, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\ps_modules\RemoteDeployer\RemoteDeployer.psm1: line 134
##[debug]at Copy-FilesParallellyToAzureVMs, D:\a\_tasks\AzureFileCopy_eb72cb01427b-a8a1-1b31ccac8a43\4.237.1\Utility.ps1: line 871
##[debug]at Copy-FilesToAzureVMsFromStorageContainer, D:\a\_tasks\AzureFileCopye5-427b-a8a1-1b31ccac8a43\4.237.1\Utility.ps1: line 1015
##[debug]at <ScriptBlock>, D:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\4.237.1\AzureFileCopy.ps1: line 260
##[debug]at <ScriptBlock>, <No file>: line 1
##[debug]at <ScriptBlock>, <No file>: line 22
##[debug]at <ScriptBlock>, <No file>: line 18
##[debug]at <ScriptBlock>, <No file>: line 1
##[debug]Exception:
##[debug]Microsoft.PowerShell.Commands.WriteErrorException: Non-Zero exit code: '1' for ComputerName: 'IMBC001'
##[error]Non-Zero exit code: '1' for ComputerName: 'PVIMC001'
##[debug]Processed: ##vso[task.logissue type=error]Non-Zero exit code: '1' for ComputerName: 'VIMBC001'
##[debug]Processed: ##vso[task.complete result=Failed]

gusAntonis · 2024-03-26T13:52:39Z

Hello guys is there anyone acting upon this issue. This has severely impacted the deployment pipelines.

gusAntonis · 2024-03-27T07:33:37Z

All deployments have been obstructed guys, hope we understand the urgency here.

AndrewBates666 · 2024-03-27T08:29:40Z

Can we get an update on this issue please? It is causing problems with many users and it seems that a simple fix of adding the ? between the url and sas token will resolve the problem. I look forward to an update.

v-mohithgc · 2024-03-28T06:58:05Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes.
I will try to get in touch with task owners to get it approved and merged ASAP.

Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)

Local task test
Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks
Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run "npm i"

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

gusAntonis · 2024-03-28T07:22:16Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.

Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)

Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run "npm i"

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

I have reviewed the PR and testing the same in our env as well.

AndrewBates666 · 2024-03-28T07:51:19Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.
Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)
Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task
Step 1: run "npm i"
Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.
Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

I have reviewed the PR and testing the same in our env as well.

We can only test in the Hosted Agents so let us know when it will be released and we will run tests on it. Best A

v-mohithgc · 2024-03-28T07:55:06Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.
Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)
Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task
Step 1: run "npm i"
Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.
Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

I have reviewed the PR and testing the same in our env as well.

We can only test in the Hosted Agents so let us know when it will be released and we will run tests on it. Best A

once @gusAntonis confirms the changes to be working fine, I will proceed to get in touch with relevant task owners to get it merged. Ideally once merged it might take 5-7 working days to be deployed on all the regions.

gusAntonis · 2024-03-28T07:57:31Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.

Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)

Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run "npm i"

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get to approved and merged ASAP.
Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)
Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task
Step 1: run "npm i"
Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.
Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

I have reviewed the PR and testing the same in our env as well.

We can only test in the Hosted Agents so let us know when it will be released and we will run tests on it. Best A

once @gusAntonis confirms the changes to be working fine, I will proceed to get in touch with relevant task owners to be get it merged.

My agents are also azure hosted, i can give this a try to use a self hosted agent as well,
But can't we use the updated task from the org. on the azure hosted agents as well.

AndrewBates666 · 2024-03-28T08:02:23Z

@v-mohithgc Hi, versioning of the tasks seems odd to me. When you complete an update rather than adding a new task version you seem to update all versions. This means that if we see a failure we don't have any option to use another version. Logically I'd expect new versions to be added so AzureFileCopy@7 which has the new code in. We've seen several failures with AzureFileCopy over the past few months and it breaks our pipelines and we don't have a way to revert unless you do so; we've lost hours of dev/test through the breakage. Just a thought...

gusAntonis · 2024-03-28T11:43:17Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get it approved and merged ASAP.

Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)

Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task

Step 1: run "npm i"

Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.

Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

So we have few issues here,
First the path specified in the upload path needs to be corrected, another \ needs to be added before _build,

And i am not able to find the build task at the specified directory :
C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\

v-mohithgc · 2024-03-28T12:04:51Z

Hi @AndrewBates666 @gusAntonis , I have created a PR #19700 for the proposed changes. I will try to get in touch with task owners to get it approved and merged ASAP.
Meanwhile, since i am unable to repro the issue from my end, I might need anyone of you to assist here, to follow the below simple procedure to test if it fixes the issue. If the changes do work, same can be followed to unblock your case here immediately without need to wait for regular task deployment (which might take upto weeks)
Local task test Have code changes ready (checkout to this branch), navigate to task root path ie C:...\azure-pipelines-tasks Note: tfx build works only on node 8 and 10, so make sure to use node 8/10 while testing the task
Step 1: run "npm i"
Step 2: Build:

node make.js build --task AzureFileCopyV5

Step 3: Install tfx:

npm install -g tfx-cli

Step 4: Login:

tfx login
Service URL : [your task test org url]
PAT :

Step 5: Upload:

tfx build tasks upload --task-path C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\ AzureFileCopyV5

New task version will be uploaded to your org, please test the changes by running the pipeline.
Step 6: Delete: // optional and can be done after all the testing is completed.

tfx build tasks delete --task-id { }

So we have few issues here, First the path specified in the upload path needs to be corrected, another \ needs to be added before _build,

And i am not able to find the build task at the specified directory : C:\AzurePipelineTask\azure-pipelines-tasks_build\Tasks\

yes its \ _build... for some reason GH is not allowing that.

so after the build succedds, _build folder will be generated, you need to copy task path under the _build folder and then do tfx upload.

gusAntonis · 2024-03-28T14:51:33Z

@v-mohithgc
I can confirm that your changes are correct, we used the AzureFileCopy version 5 and issues on the failure on that stage are resolved,
We need to merge this PR asap guys.

lukaferlez · 2024-04-04T08:26:07Z

I see that this has been merged a week ago, however it has not been released yet (we have 5.237.11 btw.)

When can we expect a release of the fix as all deployments are currently dead?

v-mohithgc · 2024-04-04T15:07:37Z

I see that this has been merged a week ago, however it has not been released yet (we have 5.237.11 btw.)

When can we expect a release of the fix as all deployments are currently dead?

I have been informed that the new task deployment cycle was started yesterday, and latest version (i.e 5.237.12) is already out on few regions. It might take the next 2-4 working days to be rolled out on all the regions.

gusAntonis · 2024-04-04T17:17:34Z

I see that this has been merged a week ago, however it has not been released yet (we have 5.237.11 btw.)
When can we expect a release of the fix as all deployments are currently dead?

I have been informed that the new task deployment cycle was started yesterday, and latest version (i.e 5.237.12) is already out on few regions. It might take the next 2-4 working days to be rolled out on all the regions.

Thanks for the update @v-mohithgc,
Really appreciate the good work you guys are doing.
I would like to know more about the release cycles and the process how the tasks get updated, any suggestions.

martijnbrands1978 · 2024-04-10T08:01:30Z

We still have version 5.237.11 in our deployments. We use Azure Hosted agents in region West Europe. When is the fix in version 5.237.12 available in West Europe?

AndrewBates666 added the bug label Mar 25, 2024

github-actions bot added Area: Release triage labels Mar 25, 2024

v-mohithgc added p1 Priority 1 (should have immediate attention) Area:RM RM task team and removed Area: Release labels Mar 26, 2024

v-mohithgc mentioned this issue Mar 28, 2024

AzureFileCopy: fix vm copy issue #19700

Merged

2 tasks

v-mohithgc closed this as completed Apr 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG]: AzureFileCopy to VM fails #19685

[BUG]: AzureFileCopy to VM fails #19685

AndrewBates666 commented Mar 25, 2024

gusAntonis commented Mar 26, 2024

v-mohithgc commented Mar 26, 2024

AndrewBates666 commented Mar 26, 2024

gusAntonis commented Mar 26, 2024

AndrewBates666 commented Mar 26, 2024 •

edited

Loading

AndrewBates666 commented Mar 26, 2024

AndrewBates666 commented Mar 26, 2024 •

edited

Loading

gusAntonis commented Mar 26, 2024

gusAntonis commented Mar 26, 2024

gusAntonis commented Mar 27, 2024 •

edited

Loading

AndrewBates666 commented Mar 27, 2024

v-mohithgc commented Mar 28, 2024 •

edited

Loading

gusAntonis commented Mar 28, 2024

AndrewBates666 commented Mar 28, 2024

v-mohithgc commented Mar 28, 2024 •

edited

Loading

gusAntonis commented Mar 28, 2024

AndrewBates666 commented Mar 28, 2024

gusAntonis commented Mar 28, 2024

v-mohithgc commented Mar 28, 2024 •

edited

Loading

gusAntonis commented Mar 28, 2024

lukaferlez commented Apr 4, 2024

v-mohithgc commented Apr 4, 2024

gusAntonis commented Apr 4, 2024

martijnbrands1978 commented Apr 10, 2024

[BUG]: AzureFileCopy to VM fails #19685

[BUG]: AzureFileCopy to VM fails #19685

Comments

AndrewBates666 commented Mar 25, 2024

New issue checklist

Task name

Task version

Issue Description

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server Version (if applicable)

Operation system

Relevant log output

Full task logs with system.debug enabled

Repro steps

gusAntonis commented Mar 26, 2024

v-mohithgc commented Mar 26, 2024

AndrewBates666 commented Mar 26, 2024

gusAntonis commented Mar 26, 2024

AndrewBates666 commented Mar 26, 2024 • edited Loading

AndrewBates666 commented Mar 26, 2024

AndrewBates666 commented Mar 26, 2024 • edited Loading

gusAntonis commented Mar 26, 2024

gusAntonis commented Mar 26, 2024

gusAntonis commented Mar 27, 2024 • edited Loading

AndrewBates666 commented Mar 27, 2024

v-mohithgc commented Mar 28, 2024 • edited Loading

gusAntonis commented Mar 28, 2024

AndrewBates666 commented Mar 28, 2024

v-mohithgc commented Mar 28, 2024 • edited Loading

gusAntonis commented Mar 28, 2024

AndrewBates666 commented Mar 28, 2024

gusAntonis commented Mar 28, 2024

v-mohithgc commented Mar 28, 2024 • edited Loading

gusAntonis commented Mar 28, 2024

lukaferlez commented Apr 4, 2024

v-mohithgc commented Apr 4, 2024

gusAntonis commented Apr 4, 2024

martijnbrands1978 commented Apr 10, 2024

AndrewBates666 commented Mar 26, 2024 •

edited

Loading

AndrewBates666 commented Mar 26, 2024 •

edited

Loading

gusAntonis commented Mar 27, 2024 •

edited

Loading

v-mohithgc commented Mar 28, 2024 •

edited

Loading

v-mohithgc commented Mar 28, 2024 •

edited

Loading

v-mohithgc commented Mar 28, 2024 •

edited

Loading