[BUG]: Initialize containers failed intermittently with Can't issue ID_TOKEN for job in 'Pending' state on v7 VMs

[StevenGranados]

What happened?

Hi,

We recently updated our self-hosted build servers to the Dxadsv7 VM sizes and started getting the following issue in the Initialize containers job:

with the error Can't issue ID_TOKEN for job in 'Pending' state. It only happens maybe 20% of the time.

I'm not sure how to get addtional logs beyond that screenshot

Thanks

Versions

4.268.0
Ubuntu 24.04

Environment type (Please select at least one enviroment where you face this issue)

• Self-Hosted
• Microsoft Hosted
• VMSS Pool
• Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Ubuntu 24.04

Version controll system

Git

Relevant log output

[dassayantan24]

Hi @StevenGranados,
It seems OIDC token request executes before the server receives the queued timeline update indicating the job has transitioned from "Pending" to "InProgress" state.

Could you please use the AGENT_ENABLE_IMMEDIATE_TIMELINE_RECORD_UPDATES knob in your pipeline YAMLl? When enabled, the agent immediately sends job state updates to the server instead of queuing them

variables:
  AGENT_ENABLE_IMMEDIATE_TIMELINE_RECORD_UPDATES: true

Once enabled, it can be shown in the "Initialize job" step like this:

[StevenGranados]

Thank you for the suggestion, I've implemented this now. Will feed back if the error continues. Is this something that will be fixed in a future version so there is no need for the variable?

[StevenGranados]

I'm just confirming that this did fix the problem. Thanks for your help.

[StevenGranados]

We are now getting occasionally very slow v7 agents that time out on jobs. Could this be due to this variable? It happens on both windows and linux machines

[StevenGranados]

This image show the CPU trace of a slow machine. It's a 32 core machine and 100-100/32 =96.875. This implies that one core is idle or near idle. Is this anything the agent is responsible for?