Skip to content

[BUG]: Sporadic failure of PowerShell runs after update to 4.268.0 #5515

@smusmu7

Description

@smusmu7

What happened?

We updated our agents from 3.238 to 4.268 mid february. Exactly one week later, multiple pipelines started receiving the following error:

========================== Starting Command Output ===========================
##[debug]Entering Invoke-VstsTool.
##[debug] Arguments: '-NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\agent\_work\_temp\d342dc0d-9f0b-40c6-9675-a0c695434e78.ps1'"'
##[debug] FileName: 'C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe'
##[debug] WorkingDirectory: 'D:\agent\_work\129\s'
"C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe" -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command ". 'D:\agent\_work\_temp\d342dc0d-9f0b-40c6-9675-a0c695434e78.ps1'"
An error occurred while creating the pipeline.
    + CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : RuntimeException
 
##[debug]Exit code: 1
##[debug]Leaving Invoke-VstsTool.
##[error]PowerShell exited with code '1'.
##[debug]Processed: ##vso[task.logissue type=error]PowerShell exited with code '1'.
##[debug]Processed: ##vso[task.complete result=Failed]Error detected
##[debug]Leaving D:\agent\_work\_tasks\PowerShell_e213ff0f-5d5c-4791-802d-52ea3e7be1f1\2.232.0\powershell.ps1.
Finishing: Cleaning managed agent pipeline workspace

This happens on multiple different servers and agents across a few of our collections. The scripts where this occurs are also wildly different and seem to have nothing special in common. It happens with every call to the powershell, e.g. the powershell task (Version 2.232.0) or inside other tasks (e.g. Msbuild).
I have not found a way to willingly cause the error, so testing is difficult. The event log does not provide additional information. Antivirus logs also have nothing related like blocking a powershell call.
Reinstalling the agent did not fix this. It usually works after a retry, but this now occurs at a rate where constant retries are not an option.

I let a simple Yaml pipeline run over the weekend about 200 times and the error did not occur once. A project with multiple large multi-stage classic pipelines fails every 2 or 3 tasks with a powershell call. But other pipelines in other collections with different self hosted agent - servers are also affected.
Some of the affected agent servers run Windows Server 2019, others run 2022.

Versions

Self hosted agent 4.268.0

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

Azure DevOps Server (Please specify exact version in the textbox below)

Azure DevOps Server Version (if applicable)

Azure DevOps Server 2022.2

Operation system

Windows Server 2019

Version controll system

git

Relevant log output

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions