Infra

How to utilize reusable components in Azure DevOps pipelines?

Kalle Marjokorpi

• 4 min read

I worked recently on a project where most of the Build and Release pipelines were refactored to utilize YAML and Bicep instead of using classic Azure pipelines and ARM templates. Refactoring was needed because some of the components used in the Azure pipelines weren't anymore supported by Azure DevOps.

This blog post covers a few techniques that can be used to create reusable components (YAML and Bicep) between multiple Azure DevOps pipelines. Blog post assumes that you have a basic knowledge of YAML and Bicep.

Environment overview

The environment followed a typical micro-service architecture approach with multiple API endpoints and a few UI applications. Microservices were coherently built and the chosen technology stack was identical. The starting point was really good to start thinking reusability of DevOps components.

Reusable YAML-templates 

YAML-based pipelines are used to orchestrate the Build and Release process in Azure DevOps.

YAML templates enable that you can separate Build and Release related functionalities to another file which can be reused to prevent "code" duplication. Operations that are consumed multiple times inside the main "the orchestrator" pipeline should be templated. 

Locally shared YAML-templates

Locally shared YAML-template means in this context that the template is shared only inside a specific Azure DevOps repository.

Application deployment job is one good example of an operation that must be repeated for many environments ("deploy-pipeline.yml" in the sample below). Template can be consumed in each environment with different parameters and this effectively prevents duplication.

This approach assumes that YAML templates are located in the same repository where the main pipeline "the orchestrator" exists. This is not the optimal way because deploy-pipeline.yml template is also wanted to be also from other Azure pipelines.

stages:
- template: pipelines/deploy-pipeline.yml
  parameters:
    stageTechnicalName: 'Test'
    azureSubscription: $(azureSubscriptionTest)
    deploymentResourceGroupName: 'rg-test-application'
    appserviceResourceName: 'app-test-my-application'
    location: $(location)
- template: pipelines/deploy-pipeline.yml
  parameters:
    stageTechnicalName: 'QA'
    azureSubscription: $(azureSubscriptionQa)
    deploymentResourceGroupName: 'rg-qa-application'
    appserviceResourceName: 'app-qa-my-application'
    location: $(location)
- template: pipelines/deploy-pipeline.yml
  parameters:
    stageTechnicalName: 'Prod'
    azureSubscription: $(azureSubscriptionProd)
    deploymentResourceGroupName: 'rg-prod-application'
    appserviceResourceName: 'app-prod-my-application'
    location: $(location)

Globally shared YAML-templates

Globally shared YAML-template enables that template to be located in a centralized Azure DevOps repository where application-specific Azure pipelines can consume templates. The diagram below illustrates that pipelines for Applications X and Y are consuming globally shared templates from separate YAML-repository.

Consumption of globally shared YAML templates is easy to configure:

1) Create a new Azure DevOps repository for globally shared YAML-templates. I have used a repository name called "Yaml.Templates" in the sample below.

2) Commit YAML templates to this new repository

3) Open application-specific main pipeline and add Resources (multi-repository configuration)

4) Add a multi-repository name to the template reference ('pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates')

resources:
  repositories:
  - repository: GloballySharedYamlTemplates
    type: git
    name: AdoProject/Yaml.Templates
    ref: main

stages:
- template: 'pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates'
  parameters:
    stageTechnicalName: 'Test'
    azureSubscription: $(azureSubscriptionTest)
    deploymentResourceGroupName: 'rg-test-application'
    appserviceResourceName: 'app-test-my-application'
    location: $(location)
- template: 'pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates'
  parameters:
    stageTechnicalName: 'QA'
    azureSubscription: $(azureSubscriptionQa)
    deploymentResourceGroupName: 'rg-qa-application'
    appserviceResourceName: 'app-qa-my-application'
    location: $(location)
- template: 'pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates'
  parameters:
    stageTechnicalName: 'Prod'
    azureSubscription: $(azureSubscriptionProd)
    deploymentResourceGroupName: 'rg-prod-application'
    appserviceResourceName: 'app-prod-my-application'
    location: $(location)

Reusable Bicep-modules

Bicep is used to configure/determine Azure infrastructure which is provisioned during deployment. The Bicep module is a similar concept to YAML-templates which enable encapsulation details to another file.

Bicep enables you to organize deployments into modules. A module is a Bicep file (or an ARM JSON template) that is deployed from another Bicep file. With modules, you improve the readability of your Bicep files by encapsulating complex details of your deployment. You can also easily reuse modules for different deployments. Source

Locally shared Bicep-modules

Locally shared Bicep modules are files inside specific Azure DevOps repositories. This is a sample of how local App Service-related Bicep-module is consumed from the main file. This App Service module is used to create an Azure App Service resource with preferred network settings. 

module appServiceModule './appservice.bicep' = {
  name: 'appServiceModule'
  params: {
      appServiceName: appServiceName
      appServicePlanName: appServicePlanName
      appServicePlanPlatform: appServicePlanPlatform
      appServicePlanSku: appServicePlanSku
      tags:defaultTags
      appSettings: appSettings
      alwaysOnEnabled: alwaysOnEnabled
      linuxFxVersion: 'DOTNETCORE|6.0'
      httpsRedirectionEnabled: false
  }
}

We want to consume this same App Service Bicep module from all Azure pipelines so we need to make it globally shared.

Globally shared Bicep-modules

Bicep supports that global Bicep modules can be shared via Azure Container Registry which acts as a centralized repository. You can find detailed information about how to configure Azure Container Registry from here.

Bicep modules can be published to Azure Container Registry with the following command. 

az bicep publish --file appservice.bicep --target br:mycontainerreqistry.azurecr.io/bicep/modules/appservice:v1

After publishing Bicep-modules from Azure Container Registry can be consumed like this

module appServiceModule 'br:mycontainerreqistry.azurecr.io/bicep/modules/appservice:v1' = {
  name: 'appServiceModule'
  params: {
      appServiceName: appServiceName
      appServicePlanName: appServicePlanName
      appServicePlanPlatform: appServicePlanPlatform
      appServicePlanSku: appServicePlanSku
      tags:defaultTags
      appSettings: appSettings
      alwaysOnEnabled: alwaysOnEnabled
      linuxFxVersion: 'DOTNETCORE|6.0'
      httpsRedirectionEnabled: false
  }
}

To publish modules to a registry, you must have permission to push an image. To deploy a module from a registry, you must have permission to pull the image. For more information about the roles that grant adequate access, see Azure Container Registry roles and permissions.

Summary

Globally shared YAML templates and Bicep-modules enable a powerful way to optimize and improve maintenance of the Azure pipeline (YAML) and Infrastructure of code files (Bicep). Especially if the environment is coherent and a similar technology stack is widely used then consumption of global components creates a lot of value.

During this project, the following functionalities were created globally.

YAML-templates:

  • Build template for .NET Framework-based applications
  • Deploy template for Azure App Service application
  • Deploy template for Azure Infrastructure
  • Database migration template
  • Unit tests execution template

Bicep-modules:

  • Create an App Service with preferred network settings
  • Key Vault policy module which assigns access policies

Comments