Lessons learned when creating reusable YAML templates

I recently wrote about how to utilize reusable YAML templates in Azure DevOps. A shared source control repository for YAML templates gives great value from a re-usability point of view. This blog post concentrates more on how you should formulate YAML templates to make them as reusable as possible. 

Template hierarchy

First a few words about template hierarchy and how I have typically structured templates.

Orchestrator is an application-specific YAML-based CI/CD pipeline whose main responsibility is to orchestrate and control the build and deployment process. Basically orchestrator template contains Stages which can be different application environments where the application will be deployed. Orchestrator consumes reusable templates to execute generic operations like building, deploying, executing integration tests, or database migrations.

Consider these tips when creating reusable YAML templates

These tips are created based on my previous experience when I have created and refactored existing YAML-based pipelines.

1. Remember single responsibility principle

Create templates that have responsibility over a single part. For example, you should have a database migration template that only executes database migration nothing else. It's much easier to maintain templates when they have clear responsibilities and overall templates are also smaller.

2. Use aggregate templates

The purpose of the aggregate template is to hide logic behind one template so consumption of the template is straightforward for the consumer. The diagram below illustrates an aggregate template called "deploy.yaml" which deploys infrastructure and application at once. Deploy.yaml template itself consumes separate "deploy-infra.yaml" and "deploy-app.yaml" templates according to the single responsibility principle.

3. Use parameters to make the template reusable

Add all configuration-related things to the parameters. Do not hard code any Build Agent path etc. inside templates.

4. Pass JobName and DependsOn to template as a parameter

When JobName and DependsOn parameters are passed to the template you can easily control from the orchestrator template which is the execution order of the templates inside the stage. JobName is a technical name of the job and can be used with DependsOn to control dependency. When the DependsOn parameter is determined as an object type you can then use multiple values.

Template:

parameters: 
    - name: jobName
      type: string
    - name: jobDisplayName
      type: string
    - name: dependsOn
      type: object

jobs:
    - job: ${{parameters.jobName}}
      displayName: ${{parameters.jobDisplayName}}
      dependsOn: ${{parameters.dependsOn}}

Orchestrator:

When the template supports JobName and DependsOn parameters you can control execution order in an orchestrator like this.

stages:
    - stage: DeploymentTest
      displayName: Deploy infra, application and database migrations to test
      jobs:
          - template: "Deploy/deploy.yaml@SharedYamlTemplates"
            parameters:
                jobName: 'DeployApplicationToTest'
                JobDisplayName: 'Deploy application to test'
                enviromentName: 'TEST'
                serviceConnection: ${{variables.test_serviceConnection}}
                appServiceName: 'app-service-test'
                infraParameterFile: 'test-parameters.json'
                azureSubscriptionId: ${{variables.test_azureSubscriptionId}}
                azureRG: ${{variables.test_azureResourceGroup}}
                dependsOn: 

          - template: "DatabaseMigrations/database-migration.yaml@SharedYamlTemplates"
            parameters:
                jobName: 'DatabaseMigrationToTest'
                JobDisplayName: 'Execute database migration to test'
                enviromentName: 'TEST'
                databaseServerName: 'databaseservertest'
                databaseName: 'testdatabase'
                projectName: 'Database.Migrations'
                serviceConnection: ${{variables.test_serviceConnection}}
                dependsOn: 
                    - DeployApplicationToTest # Database migration will be executed after DeployApplicationToTest

          - template: "Tests/integration-test.yaml@SharedYamlTemplates"
            parameters:
                jobName: 'IntegrationTestsToTest'
                jobDisplayName: 'Execute integration test to test'
                dependsOn:
                    - DeployApplicationToTest # Integration tests will be executed after DeployApplicationToTest and DatabaseMigrationToTest
                    - DatabaseMigrationToTest 

5. Add IsEnabled parameter

When a template has IsEnabled parameter you can easily control from the orchestrator whether this template should be executed or not. Sometimes you might need to execute the template only when the code is committed ex. to a specific branch.

parameters: 
    - name: jobName
      type: string
    - name: jobDisplayName
      type: string
    - name: dependsOn
      type: object
    - name: isEnabled
      type: string 

jobs:
    - job: ${{parameters.jobName}}
      displayName: ${{parameters.jobDisplayName}}
      dependsOn: ${{parameters.dependsOn}}
      condition: eq(${{parameters.isEnabled}}, true)