-
Notifications
You must be signed in to change notification settings - Fork 917
Description
What happened?
We updated our agents from 3.238 to 4.268 mid february. Exactly one week later, multiple pipelines started receiving the following error:
========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '-NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\agent\_work\_temp\d342dc0d-9f0b-40c6-9675-a0c695434e78.ps1'"'
##[debug] FileName: 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe'
##[debug] WorkingDirectory: 'D:\agent\_work\129\s'
"C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\agent\_work\_temp\d342dc0d-9f0b-40c6-9675-a0c695434e78.ps1'"
An error occurred while creating the pipeline.
+ CategoryInfo : NotSpecified: (:) [], ParentContainsErrorRecordException
+ FullyQualifiedErrorId : RuntimeException
##[debug]Exit code: 1
##[debug]Leaving Invoke-VstsTool.
##[error]PowerShell exited with code '1'.
##[debug]Processed: ##vso[task.logissue type=error]PowerShell exited with code '1'.
##[debug]Processed: ##vso[task.complete result=Failed]Error detected
##[debug]Leaving D:\agent\_work\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.232.0\powershell.ps1.
Finishing: Cleaning managed agent pipeline workspace
This happens on multiple different servers and agents across a few of our collections. The scripts where this occurs are also wildly different and seem to have nothing special in common. It happens with every call to the powershell, e.g. the powershell task (Version 2.232.0) or inside other tasks (e.g. Msbuild).
I have not found a way to willingly cause the error, so testing is difficult. The event log does not provide additional information. Antivirus logs also have nothing related like blocking a powershell call.
Reinstalling the agent did not fix this. It usually works after a retry, but this now occurs at a rate where constant retries are not an option.
I let a simple Yaml pipeline run over the weekend about 200 times and the error did not occur once. A project with multiple large multi-stage classic pipelines fails every 2 or 3 tasks with a powershell call. But other pipelines in other collections with different self hosted agent - servers are also affected.
Some of the affected agent servers run Windows Server 2019, others run 2022.
Versions
Self hosted agent 4.268.0
Environment type (Please select at least one enviroment where you face this issue)
- Self-Hosted
- Microsoft Hosted
- VMSS Pool
- Container
Azure DevOps Server type
Azure DevOps Server (Please specify exact version in the textbox below)
Azure DevOps Server Version (if applicable)
Azure DevOps Server 2022.2
Operation system
Windows Server 2019
Version controll system
git