A popular way of doing this, is by using Azure DevOps, which provides comprehensive tools for planning, developing, testing, and delivering software in an agile manner. Though an Azure DevOps (ADO) environment is very easy to set up, it is all the more difficult to secure correctly. Partly, this is due to the complex permission model at your disposal for locking down an ADO environment. In this series of blog posts, we’ll have a look at some common misconfigurations we run into during our red teaming engagements, how they can be abused, and what options you have when it comes to telemetry and monitoring.

 DevOps architecture

To talk about Azure DevOps, we need to first position it within the broader realm of Microsoft services. A typical setup can be represented by the following picture.

This company has both an on-prem and cloud environment, which are connected through some type of syncing (e.g., Entra Connect). They have an Azure DevOps environment which is managed through Entra ID, and they use several build servers to build and deploy code. These build servers can be Microsoft-hosted or self-hosted, either in Azure or on-prem. As a red teamer, this should get your spidey sense tingling, and we’ll discuss pipeline abuse extensively in the next post in this blog series.

For organizations wanting to implement agile methodologies, Azure Boards is very convenient and quick to set up. A typical setup process could look like this:

1. Create a new Azure DevOps organization: this can then be accessed at dev.azure.com/<organization_name>. Usually, this is the company name.
2. Connect your organization to Entra ID: this is used to handle authorization.
3. Create new projects in your organization: a project can then be accessed at dev.azure.com/<organization_name>/<project_name>.
4. Manage access by adding Entra groups to DevOps security groups: decide who gets access to which projects and which permissions they have.

 

 Welcome to the project page at Azure DevOps. 

Summarizing, an organization in Azure DevOps is the container for several projects that share resources. A project represents a fundamental container where you can store data and source code. Going into a project, you will find the suite of tools making up Azure DevOps:

• Azure Boards: track units of work in your software project.
• Azure Repos: source-control system, can be centralized or distributed (i.e., Git).
• Azure Pipelines: build, test, and deploy with CI/CD.
• Azure Test Plans: create manual test plans and do testing.
• Azure Artifacts: create, host and share packages.

What we see in the real world is that companies want to give practically every employee access to the boards, to fully integrate with the agile lifestyle. Instead of creating different projects related to the different functions within a company, they create only one, or at best a few, projects in their ADO organization. This configuration, combined with the intricate and complicated set of permissions within ADO, means that any type of access to a company’s ADO environment can often allow you to escalate privileges and even move laterally within the domain.

The reason for this, is that an ADO environment contains much more than just code: it includes sensitive business data, such as configuration files, secrets, build artifacts, and logs. Ensuring the integrity and security of this data is crucial to prevent unauthorized access, maintain compliance, and protect the overall reliability of the development and deployment processes. Without robust security measures, the risk of breaches, data loss, and operational disruptions significantly increases.

DevOps security

First things first: for a user to access an ADO environment they require a license. There are several types of licenses, called access levels in DevOps terminology, but the most common ones we encounter are:

• Stakeholder: this is the default access level, and can be assigned to an unlimited number of users for free. It provides partial access to Azure boards and collaboration tools, but no access to code repositories.
• Basic: this provides access to most features. When creating a new organization, you can assign the basic access level to at most 5 users for free. If you want more, you gotta pay up!

Obviously, as a red teamer, basic users are the most interesting to get your hands on. But don’t let these descriptions fool you: stakeholders can still be interesting targets, since they can, for example, view pipeline variable groups, which might contain juicy secrets. Typically, the developers will have been given basic access and other employees will get stakeholder access. From now on, we’ll assume we have somehow compromised a developer account with basic access level in DevOps, who has been given the usual permissions in a project. Let’s see what that means in more detail.

Users added to DevOps are added to one or more default security groups. Security groups are assigned permissions, which either Allow or Deny access to a feature or task. Permissions are defined at different levels: individual, organization/collection, project, or object. To grant these permissions, you will require organization or project level administrative access in the ADO environment. We will not discuss the organization level groups and the different administrative roles here (since compromising such an account is less common), but instead focus on the most common built-in security groups at the project level: Readers, Contributors, and Project Administrators. When you add a user or group to a team or project, they automatically gain access to the features associated with the default access level and security group. It is important to take note of the default permissions assigned to the roles above.

To make all this permission complexity a bit easier to understand and follow, we found a helpful permission overview created by Lionel Gurret. This slightly modified overview provides a clear example where permissions are mapped at different levels, allowing for a more comprehensive understanding of how permissions are structured and applied.

 

The above picture shows a simplified overview of how permissions can be applied. The first user, due to being in the Project Collection Administrators group, is by default added to the Project Administrators group at the project level. However, for the other two users, the roles are more granular. Being in the Project Collection Valid Users group does not grant them the Project Administrators permissions at the project level. Instead, they were specifically assigned to the Reader group at the Repos level and the Contributor group at the Pipelines level, respectively.

Developers are usually added to the Contributors group, so let’s continue with these.

 

Overview of the default permissions for the Contributors security group at the DevOps project level.

 

Looking at the permissions associated with the Contributors group at the project level, you might think the list looks a bit thin, and you would be absolutely right. That is because this list is not complete: if you want to see the permissions associated to Contributors for the different tools (repos, pipelines, …) within a project, you need to look elsewhere. Taking repositories as an example, the permissions associated with the project level Contributors groups can be found here:

 

Overview of the default permissions for the Contributors security group at the DevOps All Repositories level.

 

On top of that, the permissions on each repository can be configured separately!

 

 Overview of the default permissions for the Contributors security group at the DevOps single repository level.

For pipelines, on the other hand, you have to go through a different flow to end up at the permissions assigned … I hope you can start to appreciate what a hot mess this is, and how this level of granularity is too much for most administrators that have been asked to go agile or go home.

 

 Overview of the default permissions for the Contributors security group at the DevOps single pipeline level.

For most developers, assigning them to the Contributors security group provides access to key features in Git and should ensure they can perform their job without too many access violations. From now on, let’s assume our target is a developer with the Contributor role assigned on a project level.

Mastering repos, PATs and the REST API

Of course, developers don’t want to enter their credentials every time they make a commit, and other applications and services might also need to integrate with services and resources in ADO. Because of this, there are other authentication methods besides the usual credentials used with other cloud services. In the field, we most often run into Personal Access Tokens (PATs), which should be stored through secure solutions, such as Git Credential Manager, but are often left in plaintext and unattended in home folders, repositories, network shares, and all other places we typically scour for secrets (think .git-credentials, auth.json, …). They can be used for pretty much everything (when broadly scoped), and moreover allow you to communicate with DevOps through the Azure DevOps Services REST API. Interestingly, an account by default allows access for all authentication methods, and access needs to be explicitly denied on a per-account basis.

PATs can be managed directly from the DevOps portal (you can also use the Lifecycle management API). To create a PAT, it suffices to go to your user settings.

 

 The process of creating a new personal access token.

Here, you can select the scopes for the token to authorize for your specific tasks. As you can see from the screenshot above, it is quite tempting to click on Full access, instead of going through all categories and manually selecting the correct scopes. You are not the only one thinking this, and this is exactly why you should always be on the lookout for juicy PATs lying around.

The most widely-used scopes are the following:

• Build: provides the ability to manage builds, view build logs, and trigger new builds.
• Code: grants read and write access to repositories and pull requests.
• Release: allows management and deployment of releases, including viewing and updating release pipelines.
• User Profile: provides access to read, create, update, and delete user profiles and manage user metadata within the DevOps environment.
• Variable Groups: enables the creation and management of variable groups, including reading and writing variable group values.
• Wiki: grants permissions to read, create, edit, and manage wiki pages and related content.

For a full list of the available scopes with descriptions, you can check the docs. At the very least, the PAT usually has the Code (read) or Code (read and write) permissions; the latter granting the token read and write access to the Git repositories that the corresponding account has access to. This means that the token can perform basic Git operations, such as clone, fetch, and push, as well as create, update, and delete files in the repository. With a PAT in hand, you can use some basic PowerShell to interact with the Azure DevOps Services REST API, gather information about the user’s profile and accessible repositories, and even clone them all, so you can hunt for more secrets. Here are two basic examples of what you can achieve using a Personal Access Token to query the Azure DevOps REST API.

# Query the PAT owner user profile (User profile scope is required).
$OrganizationName="yourOrg"
$PersonalAccessToken="yourPAT"
# Base64 encode the Personal Access Token.
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$PersonalAccessToken"))
# Construct the request headers.
$headers = @{
        "Authorization" = "Basic $base64AuthInfo"
}
# Make the REST API call to get the profile.
$uri = "https://vssps.dev.azure.com/$OrganizationName/_apis/profile/profiles/me?api-version=6.0"
Invoke-RestMethod -Uri $uri -Method Get -Headers $headers

You should get some basic information like the following:

 

 The information retrieved about the user profile via the Azure DevOps REST API using PAT.

# Query accessible DevOps repositories within a project (Code (read) scope is required).
$OrganizationName="yourOrg"
$PersonalAccessToken="yourPAT"
$ProjectName="yourProject"
# Base64 encode the Personal Access Token.
$base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(":$PersonalAccessToken"))
# Make the REST API call to get the repositories.
$uri = "https://dev.azure.com/$OrganizationName/$ProjectName/_apis/git/repositories?api-version=6.0"
(Invoke-RestMethod -Uri $uri -Method Get -Headers $headers).value

This will result in the following nice list, which you can then use to clone as many repositories as your heart desires:

 

This is a good time to recall that we often see companies only using a few different projects within an organization. And since developers are typically given Contributor permission on a project level, this also means that finding a PAT (even if it is minimally scoped with Code (Read)), you will get access to all the repos within that project. More often than not, further secrets and other useful information can reveal lateral movement and privilege escalation options.

There are multiple ways to customize repositories by using branch and repository settings and policies. Repository settings and policies configure global options for all Git repositories for a project or organization, or for individual repositories. Branch policies, on the other hand, cover branch-specific controls, like requiring a pull request, a successful build, or a code review before changes can merge into a branch. Finally, repository and branch security permissions control user assignments, i.e., who can read, write, contribute to pull requests, and take other specific actions.

Microsoft has released comprehensive security best practices for DevOps, covering various aspects — including Git repositories. Key recommendations for repositories include setting policies to require a minimum number of reviewers for each pull request, configuring repository-specific security policies to enforce change management standards, and securing production secrets in dedicated Key Vaults with restricted access. Additionally, best practices, such as segregating test environments from production, disabling forking to enhance security management, and ensuring secrets are not exposed to forked builds, are highlighted. For detailed insights and implementation strategies, consult Microsoft’s security best practices documentation here.

Telemetry and monitoring

Let’s discuss visibility. Starting from the DevOps side, you want to start by ensuring the security policy to log audit events is enabled.

 

This will allow authorized users to view the audit logs.

 

This by itself is not very practical, but it is easy to create an audit stream and forward the data to your SIEM solution. Running through this process with Sentinel, for example, you will have a shiny new AzureDevOpsAuditing table to hunt with.

 

 The AzureDevOpsAuditing table in Azure Sentinel.

From a detection engineering perspective, the audit log contains a lot of information that can be expected from an audit log. The following actions are logged:

• The creation of access keys, such as PATs.
• Changes to the configuration of DevOps: this includes the creation of pipelines and mutations of user groups.

However, the day-to-day developer work leaves no traces in the logs. Thus, listing projects, listing groups, cloning code, creating branches, etc. does not end up in these logs. While there are some attack scenarios that can be detected from these logs, there seems to be a big blind spot for developer-based attack scenarios as there is no telemetry available. Another difficulty for the blue team is that the activities of an attacker and the activities of a developer can be very similar.

Besides these logs, you also have the option of using DevOps security within Microsoft Defender for Cloud.

 

DevOps security in Microsoft Defender for Cloud logs activities related to code vulnerabilities, infrastructure misconfigurations, and potential threats in CI/CD pipelines, container deployments, and cloud resources. It is meant to help identifying and remediating issues in code before they are deployed in environments, and does not seem to be intended for developer-based attack scenarios we have discussed so far.

Conclusion

Azure DevOps security is still in its early days, for attackers and defenders alike. In this first blog post, we looked at some ways of gathering information from Azure DevOps through repositories, using Personal Access Tokens and the Azure DevOps Services REST API. At the moment, it seems attackers have the edge, because of the lack of sufficient logs and the (temporary?) immaturity of Microsoft Defender for Cloud. In the coming posts, we’ll dive more into pipeline abuse and lateral movement opportunities through agent pools, which can allow attackers to pivot from the cloud all the way down to on-prem. See you then!