This blog post is part of the Azure Spring Clean 2026 virtual community event, promoting well-managed Azure tenants.

The end (of annual SSL/TLS Certificate renewals) is nigh! The maximum validity of SSL/TLS certificates is being significantly reduced over the next few years, so there’s never been a better time to automate your certificate management. In Azure TLS certificates can be used in lots of different places, some of which are easier to manage than others, and for some services you may already have a degree of automation in place. But by 2029 TLS certificates will have a maximum lifetime of 47 days, so automation will be essential to ensure you can continue to keep services running without disruption and significant overhead on the team (or teams) that maintain your infrastructure.

In this blog post we’ll explore some of the different areas of Azure that utilise TLS certificates and how you might consider automating them.

The reduction of the maximum valid lifetime of TLS certificates is being implemented in the following phases:

• Until March 14th 2026: Max 398 days (This is/was effectively 13 months, and has been the standard since September 2020)
• From March 15th 2026: Max 200 days (6 months + 1/2 of a 30-day month + 1 day)
• From March 15th 2027: Max 100 days (3 months + 1/4 of a 30-day month + 1 day)
• From March 15th 2029: Max 47 days (31 days + 1/2 of a 30-day month + 1 day)

This means that by 2029 TLS certificates will need to be renewed nearly 8 times per year (or if you don’t want to leave it until the last minute, more than likely: monthly). What’s also worth noting is that if you renew your TLS certificates before the March 14th 2026 deadline (and get the maximum 398 days), there will be no value in renewing those certificates again until after October 2026, as you’ll otherwise get a certificate with a shorter validity than you already have.

If this isn’t something you’ve thought much about up to now, a good first step would be to document everywhere in your infrastructure that currently uses TLS certificates. Some of the resource types to consider include:

• App Service
• Application Gateway
• Front Door
• AKS Ingress
• API Management
• Virtual Machines (VMs) or ScaleSets (VMSS) running webserver technologies such as IIS, NGINX or Apache Tomcat
• Service Fabric
• CDN
• Traffic Manager
• Container Apps / Instances
• Event Grid

Note: This is not an exhaustive list.

Automating the generation of your certificates will depend on your certificate provider. There is a standard protocol for certificate renewal automation called ACME (Automated Certificate Management Environment) and a number of tools that can be implemented to automate the process with a variety of providers (including Let’s Encrypt, which provides free public TLS certificates). Some certificate providers are offering their own automation platforms, but you could also consider one or more of these tools:

• Posh-ACME: A PowerShell module and ACME client to create publicly trusted SSL/TLS certificates from an ACME capable certificate authority such as Let’s Encrypt.
• win-acme: A Windows-focused ACME client (WACS) that can be scripted to manage certificate renewal, particularly for IIS and Azure-hosted workloads.
• az-acme: A specialized CLI designed to integrate ACME issuers (like Let’s Encrypt) directly with Azure Key Vault, storing certificates and enabling auto-rotation.
• EZCA: A dedicated Azure-native tool for automating certificate renewal across various Azure services, including support for Key Vault auto-rotation.
• Certbot: While general-purpose, Certbot can be used with hook scripts to renew certificates and push them to Azure resources.

These tools not only automate the request and retrieval of a certificate, but can automate the process of domain validation, such as adding a token value as a TXT DNS entry to the domain to prove ownership. They can also automate some of the deployment and configuration scenarios, such as updating the certificate binding in IIS. Most of these tools are free and open source, so well worth consideration before you inadvertently reinvent the wheel.

Another tool worth considering is haveibeenexpired.com which you can use to monitor your publicly accessible domains for expiring certificates. When you automate a process, you should also monitor it (trust, but verify).

While the certificate automation tools listed above offer a variety of features, you don’t necessarily need them. For most Azure resources, you just need to hook up to a Key Vault, and then have a scheduled process in place to renew the certificate within that, which can be relatively simple.

Azure Key Vault

Azure Key Vault is the recommended resource to use in Azure for provisioning, managing and deploying TLS certificates. Many other resources in Azure are designed to consume certificates directly from Key Vault, and a single Key Vault can be used to provide certificates to multiple resources.

You can deploy a Key Vault via Bicep as follows:

param location string = resourceGroup().location
param keyVaultName string = 'kv-example'

resource keyVault 'Microsoft.KeyVault/vaults@2023-02-01' = {
  name: keyVaultName
  location: location
  properties: {
    tenantId: subscription().tenantId
    sku: {
      family: 'A'
      name: 'standard'
    }
    enableRbacAuthorization: true
  }
}

There are a few ways to get your certificates in to Key Vault:

1. Import a TLS certificate you’ve purchased from an external provider
2. Generate a CSR (Certificate Signing Request) via Azure Key Vault which you can then use to complete a certificate request via an external provider
3. Use one of the partnered CA providers (Digicert or GlobalSign) to generate and import the certificate directly. This requires having an account with either Digicert or Globalsign and providing your credentials for these within the Key Vault.

When you upload a certificate into Key Vault, it imports various properties including the valid from and valid to dates, which you can then use to monitor the validity of your certificates (such as by configuring alerts in Azure Monitor). When a certificate is generated by Key Vault, you configure an email notification to be sent when a certain percentage of the certificates lifetime has been consumed. You can setup the same for imported certificates, by configuring “Certificate Contacts” on the Key Vault, and going to the Issuance Policy settings of the certificate to configure the percentage you want to be notified at. By default this is 80%.

When you use one of the partnered CA (Certificate Authority) providers, Key Vault refers to this as using an Integrated CA and you can configure automatic renewal. If you’re not using an Integrated CA, then you’ll need to automate the process of retrieving a new certificate, and then upload that into Key Vault.

You can do the latter pretty easily via PowerShell:

# Variables
$kv   = "my-keyvault"
$name = "my-certificate"
$pfx  = "C:\certs\mycert.pfx"
$pw   = ConvertTo-SecureString "PfxPasswordHere" -AsPlainText -Force

# Import certificate
Import-AzKeyVaultCertificate -VaultName $kv -Name $name -FilePath $pfx -Password $pw

Ensure you use the same Name value when renewing the certificate and a new version will be uploaded to the existing entry in the Key Vault.

App Service

Azure App Service allows you to create a free managed TLS certificate (provided by Digicert), which is then fully managed by the App Service and automatically renewed prior to expiry. If you instead provide your own TLS certificate, you can either upload this directly into App Service, or import it from a Key Vault. The latter is recommended, as its then easier (per the above) to monitor and manage your certificate. When you update the certificate entry in Key Vault, App Service automatically syncs the new version within 24 hours and without downtime. You can also trigger a manual sync in App Service.

Note when targetting the certificate in Key Vault you must use the non-version specific URL. Doing so will ensure it always pulls the latest certificate.

If you upload the certificate directly to App Service, you will need to modify the certificate in the App Service config, either via the Portal, or via infrastructure code such as ARM or Bicep. Managing the certificate in Key Vault decouples the management of the certificate from the configuration of the resource.

Here’s an example of how you might configure an App Service to use a Key Vault certificate via Bicep:

param location string = resourceGroup().location
param appName string = 'myapp-${uniqueString(resourceGroup().id)}'
param kvName string = 'kv-example'
param certName string = 'mySslCert'
param customHostname string = 'www.mycustomdomain.com'

resource appServicePlan 'Microsoft.Web/serverfarms@2023-01-01' = {
  name: 'asp-${appName}'
  location: location
  sku: {
    name: 'P1v2'
    tier: 'PremiumV2'
  }
}

resource kv 'Microsoft.KeyVault/vaults@2023-02-01' existing = {
  name: kvName
}

resource webApp 'Microsoft.Web/sites@2023-01-01' = {
  name: appName
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    serverFarmId: appServicePlan.id
    httpsOnly: true
  }
}

resource certificate 'Microsoft.Web/certificates@2023-01-01' = {
  name: certName
  location: location
  properties: {
    keyVaultId: kv.id
    keyVaultSecretName: certName
    serverFarmId: appServicePlan.id
  }
}

If you reference the certificate from Key Vault, you need to grant the App Service permission to read the certificate via the Key Vault Secrets User role. The simplest way to grant this is to configure the App Service to have a System Assigned Identity and then grant this Identity this permission on the Key Vault.

Application Gateway

Similar to App Service, you can implement a TLS certificate in Application Gateway by either configuring it directly on the resource, or via a Key Vault secret. As before, I recommend using a Key Vault to decouple management of the certificate from configuration of the resource.

Here’s a partial example of how you might configure an Application Gateway to use a certificate from a Key Vault via Bicep:

param location string = resourceGroup().location
param appGwName string = 'agw-example'
param keyVaultName string = 'kv-example'
param certSecretName string = 'appgw-cert'
param vnetName string = 'agw-vnet'
param subnetName string = 'agw-subnet'

resource keyVault 'Microsoft.KeyVault/vaults@2023-02-01' existing = {
  name: keyVaultName
}

resource appGateway 'Microsoft.Network/applicationGateways@2023-05-01' = {
  name: appGwName
  location: location
  identity: {
    type: 'UserAssigned'
  }

  properties: {
    sku: {
      name: 'Standard_v2'
      tier: 'Standard_v2'
      capacity: 2
    }

    gatewayIPConfigurations: [
      {
        name: 'gwIpConfig'
        properties: {
          subnet: {
            id: resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, subnetName)
          }
        }
      }
    ]

    frontendIPConfigurations: [
      {
        name: 'frontendIp'
        properties: {
          publicIPAddress: {
            id: resourceId('Microsoft.Network/publicIPAddresses', 'agw-pip')
          }
        }
      }
    ]

    frontendPorts: [
      {
        name: 'httpsPort'
        properties: {
          port: 443
        }
      }
    ]

    sslCertificates: [
      {
        name: 'sslCertFromKV'
        properties: {
          keyVaultSecretId: 'https://${kvName}.vault.azure.net/secrets/${certSecretName}'
        }
      }
    ]

    httpListeners: [
      ..
    ]
  }
}

Front Door

The pattern for Azure Front Door is very similar to App Service and App Gateway when referencing a certificate from a Key Vault. As before, the System Identity of the Front Door will require the Key Vault Secrets User role on the Key Vault.

Here’s a Bicep example for configuring Azure Front Door with a certificate from Key Vault:

param profileName string = 'afd-profile'
param keyVaultName string = 'kv-prod-network'
param certName string = 'site-cert'
param secretName string = 'site-cert-secret'

resource kv 'Microsoft.KeyVault/vaults@2023-02-01' existing = {
  name: keyVaultName
}

resource afdProfile 'Microsoft.Cdn/profiles@2023-05-01' existing = {
  name: profileName
}

resource afdSecret 'Microsoft.Cdn/profiles/secrets@2023-05-01' = {
  name: '${afdProfile.name}/${secretName}'
  properties: {
    parameters: {
      type: 'CustomerCertificate'
      secretSource: {
        id: kv.id
      }
      secretVersion: ''
      secretName: certName
    }
  }
}

resource customDomain 'Microsoft.Cdn/profiles/customDomains@2023-05-01' = {
  name: '${afdProfile.name}/mydomain'
  properties: {
    hostName: 'www.example.com'
    tlsSettings: {
      certificateType: 'CustomerCertificate'
      minimumTlsVersion: 'TLS12'
      secret: {
        id: afdSecret.id
      }
    }
  }
}

AKS Ingress

For Azure Kubernetes Service, you can also reference a certificate from a Key Vault. However there’s slight additional complexity here in that AKS cannot read from the vault directly. Instead you need to install the CSI Driver in Kubernetes, so that it can then read from the Key Vault and implement the certificate as a Kubernetes TLS secret. Once again using the System Assigned Identity of AKS is recommended for granting access to the Key Vault certificate via the Key Vault Secrets User role.

The CSI Driver can be installed via Helm:

helm repo add csi-secrets-store-provider-azure https://azure.github.io/secrets-store-csi-driver-provider-azure/charts
helm install csi csi-secrets-store-provider-azure/csi-secrets-store-provider-azure

And then within the Kubernetes config you define a SecretProviderClass:

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: tls-cert-provider
spec:
  provider: azure
  parameters:
    keyvaultName: kv-prod-network
    objects: |
      array:
        - |
          objectName: site-cert
          objectType: secret
    tenantId: <tenant-id>

And sync the secret to Kubernetes:

secretObjects:
- secretName: ingress-tls
  type: kubernetes.io/tls
  data:
  - objectName: site-cert
    key: tls.key
  - objectName: site-cert
    key: tls.crt

Finally you configure the Ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: app-ingress
spec:
  tls:
  - hosts:
    - www.example.com
    secretName: ingress-tls
  rules:
  - host: www.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: webapp
            port:
              number: 80

API Management

API Management is similar to the other services in that you can upload a TLS certificate to it directly, or reference one in an Azure Key Vault. Once again the latter is the recommendation, for the same reasons as previously stated: you separate the maintenance of the TLS certificate from the configuration of the resource. Again the API Management resource needs permission to read the certificate from the Key Vault, which can be implemented by configuring it with a System Assigned Identity and then giving that the KeyVault Secrets Reader role on the Key Vault.

When you update the certificate in the KeyVault API Management should detect the change within 4 hours and then apply it within 20 minutes. As with the other resources, it is important to reference the Key Vault secret for the certificate without a specific version, so that it always retrieves the latest.

param apimName string
param location string = resourceGroup().location
param gatewayHostname string
param keyVaultSecretId string

resource apim 'Microsoft.ApiManagement/service@2023-05-01-preview' existing = {
  name: apimName
}

resource apimCustomDomain 'Microsoft.ApiManagement/service@2023-05-01-preview' = {
  name: apim.name
  location: location

  properties: {
    hostnameConfigurations: [
      {
        type: 'Proxy'
        hostName: gatewayHostname

        keyVaultId: keyVaultSecretId

        defaultSslBinding: false
        negotiateClientCertificate: false
      }
    ]
  }
}

Virtual Machines or ScaleSets

You can often avoid having TLS certificates on VMs at all, by terminating TLS earlier in the network such as at an Application Gateway, Front Door or Load Balancer. However if you have a need to configure a TLS certificate on a Virtual Machine or ScaleSet there are a couple of good options.

One approach is to use the Azure Key Vault VM Extension. This gives you a very similar pattern as with the other resources mentioned earlier. You reference the non-version specific URL of the certificate’s secret. The extension polls at a specified interval for changes to this secret, and when a new version is detected it downloads the certificate and installs it into the certificate store location of your choice. Here’s a Bicep example for a Windows VM (a similar configuration can be used for Linux VMs):

resource kvExtension 'Microsoft.Compute/virtualMachines/extensions@2023-09-01' = {
  name: '${vm.name}/KeyVaultForWindows'
  location: resourceGroup().location

  properties: {
    publisher: 'Microsoft.Azure.KeyVault'
    type: 'KeyVaultForWindows'
    typeHandlerVersion: '3.0'
    autoUpgradeMinorVersion: true

    settings: {
      secretsManagementSettings: {
        pollingIntervalInS: '3600'
        requireInitialSync: true
        observedCertificates: [
          {
            url: 'https://my-keyvault.vault.azure.net/secrets/my-cert'
            certificateStoreName: 'My'
            certificateStoreLocation: 'LocalMachine'
          }
        ]
      }
    }
  }

  dependsOn: [
    vm
  ]
}

If you’re using IIS, it can be configured to automatically rebind when a new certificate is detected.

Another approach is to use the certbot tool mentioned earlier. This can be installed via apt:

sudo apt update
sudo apt install certbot python3-certbot-nginx -y

And then called as follows to perform an initial retrieval of the cert:

sudo certbot --nginx -d app.example.com

This:

• Validates domain ownership
• Generates a certificate
• Updates Nginx configuration automatically.

Then you can setup a cron job to perform automatic renewal:

echo "0 0,12 * * * root /opt/certbot/bin/python -c 'import random; import time; time.sleep(random.random() * 3600)' && sudo certbot renew -q" | sudo tee -a /etc/crontab > /dev/null

There are more detailed instructions on the official website for a variety of Operating Systems.

Summary

If you’re reading this in 2026, there’s still a good amount of time to get a handle on your certificate management before the activity becomes a monthly task. When considering how to manage certificates in Azure, I recommend the following:

• Audit your estate and ensure you have a good understanding of everywhere certificates are used.
• Use the fully managed certificate services and automatic renewal where possible.
• Centralise certificate management via Key Vault, and consider minimising the number of Key Vaults used to store certificates, which reduces the number of places certificates need to be updated (if you are not able to use auto-renew). You might consider a KeyVault per environment stage for example (Dev -> Staging -> Production) that is then shared by each of the resources in that stage.
• Implement monitoring of certificates to ensure you catch expiring certificates before they become an outage.

Azure provides managed identities as a secure and convenient way to manage access to resources. By configuring a managed identity you no longer need to store secrets, keys or credentials. For SQL server, the traditional way to access a database would be via a connection string that would include a username and password. However managed identities are password-less, so eliminate the need to provide a password directly in the connection string.

The complexity of this task comes from the following issues:

• To configure a managed identity as a user within Azure SQL the user must be configured by another Entra ID account
• The SQL Server itself must be able to validate the Entra ID user you are configuring

Getting this setup to execute via a pipeline can be done via the following three tasks:

1. Configure an AD Administrator for the SQL server

To configure a managed identity as a user within Azure SQL the user must be configured by another Entra ID account

At first this might seem like a chicken and egg scenario, because how can you configure an Entra ID account as a user without already having an Entra ID account as a user? The answer is to configure an Entra ID account as the SQL Server AD Administrator. If you connect using local SQL credentials (even those with system administrator permissions) and attempt to create a SQL login for an Entra ID user you’ll get an error.

Because we wanted to be able to configure the Logic App managed identity with permissions to SQL via the pipeline, I made the service principal of the pipeline the Azure AD Administrator. I did this via PowerShell as follows (this script ran in the pipeline that deploys Azure SQL, which has new input variables to specify the name $sqlAdAdminName and object ID $sqlAdAdminObjectId of the pipeline service principal):

# Get existing Sql server configuration
$sqlServer = az sql server show --name $sqlServerName --resource-group $resourceGroup --output json | ConvertFrom-Json

# Check if AD admin already exists
$existingAdAdministrator = $sqlServer.administrators | Where-Object { $_.administratorType -eq 'ActiveDirectory' }

# Configure AD admin if not already set to the required SQL Admin name
if ($existingAdAdministrator.login -ne $sqlAdAdminName) {

    Write-Host "Setting SQL AD Admin.."
    az sql server ad-admin create --resource-group $resourceGroup --server $sqlServerName --display-name $sqlAdAdminName --object-id $sqlAdAdminObjectId

    if ($? -eq $false) {
        Write-Error "Error setting SQL AD Admin $sqlAdAdminName with objectId $sqlAdAdminObjectId."
    }
}
else {
    Write-Host "SQL AD Admin already set to $sqlAdAdminName."
}

If you didn’t want to use the pipeline’s service principal as the AD administrator, you could configure another Entra ID user or group and then provide the credentials for that user during your pipeline execution. Obviously using the pipeline’s service principal negates the need to store any credentials, but it does mean anything executed via the pipeline’s service principal in the future has AD Administrator access to the SQL Server, so would have to be carefully secured. It’s up to you to decide whether this is appropriate for your specific database/scenario.

2. Enable a system-assigned (or user-assigned) managed identity on the SQL server

The SQL Server itself must be able to validate the Entra ID user you are configuring

You might think (as I did) that the Entra ID user you’re now connecting to the SQL server with to execute your commands to create the new Entra ID login would have enough access by itself to complete the task. But actually, when you add a directory user to SQL server, it seems the SQL server itself has to do a bit of validation against the directory. To do this, the Azure SQL Server needs either a System Assigned identity, or User Assigned identity configured with specific permissions to read the Entra ID directory.

You can enable a System Assigned identity via PowerShell as follows:

# Check if existing System Assigned Identity is set
$existingSAIdentity = $sqlServer.identity | Where-Object { $_.type -eq 'SystemAssigned' }

# Configure System Assigned Identity if not already set
if (-not $existingSAIdentity) {

    Write-Host "Setting System Assigned Identity.."
    az sql server update --resource-group $resourceGroup --name $sqlServerName --assign_identity --identity-type SystemAssigned
}
else {
    Write-Host "System Assigned Identity already set."
}

Once you’ve configured a System Assigned identity you need to grant it the Directory Readers role. This has to be done by an Entra ID account with Privileged Role Administrator or higher permissions. We did this step manually, but it could be automated. Alternatively if you use a user-assigned managed identity, you can configure specific Microsoft Graph permissions for it via PowerShell as described in this Microsoft article.

3. Execute your SQL scripts to configure the managed identity permissions and roles

Having completed the above, we could now configure the Logic App managed identity as a user in the SQL server as follows (this script runs in another pipeline where the specific logic app is deployed that we wanted to grant access):

Because we configured the pipeline user as a AD Admin, the below uses Az CLI to get the current users access token to authenticate to SQL. If you configured another user as SQL Server AD Admin you’ll need to authenticate to SQL with those credentials.

# Get the access token of the current pipeline user session
$accessToken = az account get-access-token --resource https://database.windows.net/ --query accessToken --output tsv

$invokeSqlcmd = @{
    ServerInstance  = "${sqlServerName}.database.windows.net"
    AccessToken     = $accessToken
    OutputSqlErrors = $true
}

# Specify the name of the Logic App resource, which has system assigned identity enabled
$laName = "your-logic-app-resource-name"

# Check if the DB user is already configured
try {
    $dbuserAlreadyExists = Invoke-Sqlcmd @invokeSqlcmd -Query "SELECT name FROM sys.sysusers WHERE name='$laName'" -Database $dbName -ErrorAction Stop
}
catch {
    Write-Error $_
    throw "Failed to check for existing login for id $laName in database $dbname."
}

if (-not $dbuserAlreadyExists) {

    Write-Verbose "Creating user $laName in database $dbName from external provider.."
    Invoke-Sqlcmd @invokeSqlcmd -Database $dbName -Query "CREATE USER [$laName] FROM EXTERNAL PROVIDER"

    Write-Verbose "Adding role db_datareader to user $laName.."
    Invoke-Sqlcmd @invokeSqlcmd -Database $dbName -Query "ALTER ROLE [db_datareader] ADD MEMBER [$laName];"
}
else {
    Write-Verbose "User $laName already exists."
}

For more information on the above (and to ensure you’re following the latest Microsoft guidance) check out the official documentation for configuring Microsoft Entra authentication with Azure SQL.

The page on Microsoft Entra service principals with Azure SQL was also useful.

Many DevOps tools were gifted to the Engineers, who above all else, desired automation. For within these tools was bound the strength and will to govern the Cloud. But they were all of them deceived, for another tool was made. In the land of Microsoft, in the fires of Mount Azure, the Architect Bernie White forged, in open source, a tool to validate all others. And into this tool he poured his creativity, his mastery and his will to improve all infrastructure.

One tool to PSRule them all.

I suspect it was something like that anyway.

PSrule can be found in GitHub and has been in development since December 2018. The current stable version is v2.9.0 (v3 is in development). The official website has lots of guidance on getting started, and is officially described as:

“A rules engine geared towards testing Infrastructure as Code (IaC). Rules you write or import perform static analysis on IaC artifacts such as: templates, manifests, pipelines, and workflows.”

The idea is that you (or Microsoft, the community, the Council of Elrond) define rules for how your infrastructure should be configured and PSRule (executed as part of your development process) confirms those constraints are being followed. Infrastructure code is often complex, and can be developed by multiple individuals or teams over time. PSRule can act as a guard rail to ensure the infrastructure requirements of your organisation are being followed, or used to evaluated the quality of existing infrastructure.

Obviously developing these rules is itself a timely endeavour, but PSRule has done the heavy lifting for you by providing various pre-built rules based on best practice guidance such as the Azure Well-architected Framework. PSRule is extensible, so you choose which existing rules you want to use, and customise them (and/or develop your own rules) to meet your requirements.

Begin the journey

“It’s a dangerous business, Frodo, going out your door.” — Bilbo Baggins

Installing PSRule for Azure requires no difficult journey to Mount Doom. There is a dedicated site for this module, with its own Getting started page. You can install it directly, as follows:

• Install PowerShell 7 (if you haven’t already)
• Install PSRule (with the pre-built Azure rules) from the PowerShell Gallery:

Install-Module -Name 'PSRule.Rules.Azure' -Repository PSGallery -Scope CurrentUser

PSRule for Azure is officially described as:

“A pre-built set of tests and documentation to help you configure Azure solutions. These tests allow you to check your Infrastructure as Code (IaC) before or after deployment to Azure. PSRule for Azure includes unit tests that check how Azure resources defined in ARM templates or Bicep code are configured.”

As it turns out, one does not simply test Terraform with PSRule.

But it’s not that complicated. At the moment you can’t perform static analysis of your Terraform files (or Terraform plan output), but you can indicate your support for the feature here. For now, you need to deploy resources to Azure, run a command to export the configuration of those resources to a JSON file, which PSRule can then analyse. While this doesn’t make PSRule as useful as it is for analysing Bicep (for which there is also a VSCode extension so you can get feedback while developing), if you deploy your infrastructure through a series of environments for testing (or even if you don’t) I can see PSRule still forming a valuable part of that pipeline.

To summarise, here’s the steps to take to analyse Terraform resources:

1. Deploy your Terraform code to Azure (ideally to a non-production environment).
2. Ensure you have authenticated to Azure via Connect-AzAccount and have the relevant subscription selected.
3. Export your Azure resources to a specified directory (and ensure the directory exists):

New-Item -Path out -ItemType Directory
Export-AzRuleData -OutputPath "$pwd/out"

By default this will export all of the resources for the currently selected subscription. You can use -Subscription to specify one or more subscriptions, or -ResourceGroupName to specify one or more resource groups. You can also use -Tag to export resources with one or more specified tags.

If you want to go big (rather than return to the shire), you can use -All to export resources from all the subscriptions your current context can access.

The output file/s are named with the guid of the subscription. This is true even if you filter to specific resources by Resource Group or Tag, and any existing file with the same name will be overwritten.

Face your demons

“It is the small things, everyday deeds of ordinary folk that keep the darkness at bay.” — Gandalf

It’s now time to analyse the output. You do this with Invoke-PSRule which you need to point at your export file/s and the PSRule.Rules.Azure module:

Invoke-PsRule -InputPath "$pwd/out/" -Module 'PSRule.Rules.Azure'

Depending on your resources, you’ll probably get quite a lot of output because by default the tool returns all results (pass, fail and error). If you want to just see failed results, execute:

Invoke-PSRule -InputPath "$pwd/out/" -Module 'PSRule.Rules.Azure' -Outcome Fail

Results from PSRule are returned as PowerShell objects, so it’s worthwhile returning them to a variable as there are more properties returned than you see in the default view above. But what you do get by default is the rule outcomes grouped by resource, with the name of the rule and a further description of how to address it in the Recommendation.

Further properties in the object that are useful include:

• Ref — This is the unique reference number for the rule. You can use this or the rule name to get to the relevant reference page which has more detail explaining the rule and sometimes examples of how to address it.
• Reason — This is useful to get more specific detail of the violation, particularly if it relates to one or more sub-resources.

You also might want to export the results for some further/offline analysis and PSRule makes this easy with it’s -OutputPath parameter, along with -OutputFormat that can be used to output to various formats such as CSV, JSON, Markdown, YAML and NUnit3.

There are some other switches on the command that are worth exploring, one of which is -As Summary which simply returns a table showing for each rule how many resources passed and failed:

RuleName                            Pass  Fail  Outcome
--------                            ----  ----  -------
Azure.AppService.PlanInstanceCount  0     4     Fail
Azure.AppService.MinTLS             20    0     Pass
Azure.KeyVault.Logs                 19    0     Pass
Azure.KeyVault.Name                 19    0     Pass
Azure.Resource.UseTags              153   0     Pass
Azure.Resource.AllowedRegions       154   0     Pass
Azure.ResourceGroup.Name            20    0     Pass
Azure.ServiceBus.Usage              4     0     Pass
Azure.SQL.FirewallRuleCount         1     1     Fail
Azure.SQL.AllowAzureAccess          0     2     Fail
Azure.Storage.UseReplication        23    6     Fail
Azure.Storage.SoftDelete            2     12    Fail
Azure.RBAC.UseRGDelegation          20    0     Pass
Azure.VM.PublicIPAttached           5     1     Fail
Azure.VNET.UseNSGs                  0     4     Fail
...

Make the hard choices (or easy ones)

“This task was appointed to you. And if you do not find a way, no one will.” — Galadriel

Once you have the output there’s obviously two ways forward (three if you count doing nothing):

1. Address the failure by modifying your Terraform code
2. Exclude the rule

If there are issues that you can and want to fix, then simply make the required changes in your Terraform code (using the guidance provided by the recommendation). Redeploy your infrastructure and re-run the export of the rule data and analysis to confirm the test now passes.

There may be issues that you don’t want to fix, because the supposed misconfiguration is appropriate for your infrastructure (for example, you might have a Storage account that you need to be publicly accessible) or there may be entire rulesets that you don’t consider in scope for your infrastructure (you may have no need for tags for example). PSRule allows you to control these exceptions by configuring exclusion or suppression.

These customisations are configured by creating a file called ps-rule.yaml.

You can exclude a rule as follows:

rule:
  exclude:
    # Ignore the following rules for all resources
    - Azure.Resource.UseTags

And you can suppress a rule for specific resources like this:

suppression:
  Azure.Storage.BlobPublicAccess:
    # Ignore blob public access on the following storage account
    - mypublicstorageaccount

These rules will no longer pass or fail.

There’s some further cleverness you can do to avoid having to manually populate resources in these files, by instead creating logic that sets up exclusions based on the value of fields such as names or tags. These are called suppression groups.

The built-in rules are routinely updated (usually monthly). As the rules are updated, the output you get may change. You can manage this by running PSRule against a specified baseline which are published quarterly. This keeps your testing consistent, until you decide to move to a new baseline. There are also pillar specific baselines, for example if you’re only interested in using PSRule to evaluate cost optimisation you can use the Azure.Pillar.CostOptimization baseline. You can also create your own custom baselines.

The built-in baselines are listed here. To run PSRule against a specified baseline you use the -Baseline parameter:

Invoke-PSRule -InputPath "$pwd/out/" -Module 'PSRule.Rules.Azure' -Outcome Fail -Baseline 'Azure.GA_2024_09'

Note: you will see a warning if you use a baseline that is outdated (i.e, not the latest).

Don’t go it alone

“Help me bear this burden.” — Frodo

While it might be helpful to have a band of hobbitses to assist with this work, a more DevOpses approach might be to implement a pipeline. Once you’ve done the above: established your baseline and excluded or suppressed rules that do not apply to your infrastructure, it may make sense to include a run of PSRule as part of your CI pipeline, and perhaps as a gate for infrastructure Pull Requests. PSRule provides guidance on this but it is once again geared towards testing Bicep or ARM static files. For Terraform we need to do an in-flight analysis post-deployment.

Here’s how you might do that in Azure DevOps:

trigger:
- main

pool:
  vmImage: 'ubuntu-latest'

stages:
- stage: Plan
  displayName: 'Terraform Plan'
  jobs:
  - job: TerraformPlan
    displayName: 'Terraform Plan'
    steps:
    - task: TerraformInstaller@0
      inputs:
        terraformVersion: 'latest'

    - script: |
        terraform --version
        terraform init -backend-config="storage_account_name=$(storageAccountName)" -backend-config="container_name=$(containerName)" -backend-config="key=$(key)" -backend-config="access_key=$(accessKey)"
        terraform plan -out=tfplan
      displayName: 'Terraform Init and Plan'

- stage: Deploy
  displayName: 'Terraform Apply'
  jobs:
  - job: TerraformApply
    displayName: 'Terraform Apply'
    steps:
    - task: TerraformInstaller@0
      inputs:
        terraformVersion: 'latest'

    - script: |
        terraform --version
        terraform init -backend-config="storage_account_name=$(storageAccountName)" -backend-config="container_name=$(containerName)" -backend-config="key=$(key)" -backend-config="access_key=$(accessKey)"
        terraform apply -auto-approve tfplan
      displayName: 'Terraform Init and Apply'

- stage: PSRule
  displayName: 'Execute PSRule'
  jobs:
  - job: PSRule
    displayName: 'Execute PSRule'
    steps:
      - task: AzurePowerShell@5
            displayName: Export PSRule Data and Execute Rules
            inputs:
              azureSubscription: $(azureSubscription)
              scriptType: InlineScript
              azurePowerShellVersion: LatestVersion
              Inline: |
                Install-Module -Name 'PSRule.Rules.Azure' -Scope CurrentUser -Force -ErrorAction Stop

                New-Item -Path out -ItemType Directory -Force
                Export-AzRuleData -OutputPath 'out'

                Assert-PSRule -InputPath 'out' -Module 'PSRule.Rules.Azure' -Outcome Fail -Baseline 'Azure.GA_2024_12'

Note the usage of Assert-PSRule instead of Invoke-PSRule. This command returns formatted text instead of PowerShell objects which is easier to read in the output of a pipeline job.

The pipeline job will fail if the tests fail, and the output will look something like this:

Journeying onward

“Don’t adventures ever have an end? I suppose not. Someone else always has to carry on the story.” — Bilbo Baggins

If you’ve conquered all of the above, well done! But what comes after? Well you could give some consideration to authoring your own custom rules. To do this:

1. Create a .ps-rule directory in your repository.
2. Create a new custom rule ps1 file. The rule name is up to you, but must be unique. For example: Org.Azure.Rule.ps1.
3. In the file define your rule. For example, here’s how you might configure a rule for custom tags:

# Synopsis: Resource Groups must have all mandatory tags defined.
Rule 'Org.Azure.RG.Tags' -Type 'Microsoft.Resources/resourceGroups' {
    $hasTags = $Assert.HasField($TargetObject, 'Tags')
    if (!$hasTags.Result) {
        return $hasTags
    }

    # Require tags be case-sensitive
    $Assert.HasField($TargetObject.tags, 'costCentre', $True)
    $Assert.HasField($TargetObject.tags, 'env', $True)
}

In the example above the HasField assertion takes 3 inputs, the value being evaluated, the expected value and a boolean indicating whether the comparison should be case-sensitive. There are plenty of other assertions you can use.

Before your rule will work, you will also need to add the following to the ps-rule.yaml file we created earlier:

# Configure binding options
binding:
  targetType:
    - 'resourceType'
    - 'type'

This binding allows custom rules to use the -Type parameter. The in-built rules detect this automatically, but custom rules check the ps-rule.yaml file for their type binding configuration.

You can test your custom rule by executing Invoke-PSRule with a -Path parameter:

Invoke-PSRule -Path "$pwd/.ps-rule/" -InputPath "$pwd/out/"

The output will look something like this:

RuleName                            Outcome    Recommendation
--------                            -------    --------------
Org.Azure.RG.Tags                   Fail

   TargetName: sometargetresource

You’ll notice there’s no Recommendation returned. We can add this by adding the Recommend keyword into your rule:

Rule 'Org.Azure.RG.Tags' -Type 'Microsoft.Resources/resourceGroups' {
    $hasTags = $Assert.HasField($TargetObject, 'Tags')
    if (!$hasTags.Result) {
        return $hasTags
    }

    # Require tags be case-sensitive
    $Assert.HasField($TargetObject.tags, 'costCentre', $True)
    $Assert.HasField($TargetObject.tags, 'env', $True)

    Recommend "The following tags are mandatory: 'costCentre','env'"
}

And now we can see the message:

RuleName                            Outcome    Recommendation
--------                            -------    --------------
Org.Azure.RG.Tags                   Fail       The following tags are mandatory: 'costCentre','env'

   TargetName: sometargetresource

I hope you’ve found this a useful introduction to PSRule. While the focus of this blog was on Terraform, most of what is detailed above can be used to test infrastructure deployed via any mechanism (even resources created manually, but please don’t :) ).

For other great content in this year’s Azure Spring Clean, check out the website or follow the Azure Spring Clean list on BlueSky.

Did you know that in the books, 17 years pass between Frodo receiving the ring and heading off on his quest? Once he finally got going he travelled about 1800 miles, from Bag End to Mount Doom, in about 6 months. It would likely have been quicker by eagle.

If no one knows what you're doing, what's the point? pic.twitter.com/JdNcIDpzzU

— Doug Finke (@dfinke) June 6, 2024

Which got me thinking about the modules I’ve published in GitHub and the PowerShell Gallery over the last few years (almost 10 in fact). So I’ve decided to revisit them, why they exist, and whether they are still useful, in case it inspires anyone else to go on their own decade long journey of publishing things to the gallery.

If you’ve never published a module to the PowerShell Gallery and are interested in how to get started, there’s a detailed guide in the official documentation here.

I thought it might be interesting to do that in order of popularity, and (if you’ve used a consistent author name) it’s actually quite easy (although not entirely quick as I think it has to parse every module) to get your modules from the gallery along with their download counts by using Find-Module. For example:

function Find-ModuleByAuthor {
  [cmdletbinding()]
  param(
    [parameter(Mandatory)]
    $Author
  )

  Write-Verbose "Finding all modules by $Author.."
  $Modules = Find-Module | Where-Object { $_.Author -eq $Author }

  foreach ($Module in $Modules) {

    Write-Verbose "Finding all versions of $($Module.Name).."
    $ModuleVersions = Find-Module -Name $Module.Name -AllVersions

    $FirstPublishedDate = ($ModuleVersions | Sort { $_.Version -as [version] } | 
      Select -First 1).PublishedDate
    $DownloadCount = [int]($ModuleVersions.AdditionalMetadata.versionDownloadCount | 
      Measure-Object -Sum).Sum

    [PSCustomObject]@{
      Name               = $Module.Name
      Downloads          = $DownloadCount
      Description        = $Module.Description
      Link               = $Module.ProjectUri
      Published          = $FirstPublishedDate
      LastUpdated        = $Module.PublishedDate
    }
  }
}

Here’s where I discovered a horrifying statistic:

Modules I’ve published in the PowerShell Gallery have been downloaded (in total) almost a million times.

(well.. 963,850 times to be exact):

$Modules = Find-ModuleByAuthor -Author 'Mark Wragg'
$Modules | Select Downloads,Published,Name,Link | Sort Downloads

($modules.downloadCount | Measure -Sum).Sum

963850

So this is a story all about how (my life got flipped-turned upside down, and I’d like to take a minute just sit right there while I tell you how) I’m about to become a PowerShell Gallery millionaire.

AzCostTools

• First published: 14 Jan 2024
• Last updated: 02 Jun 2024
• Download count: 359

AzCostTools is one of my newest projects. I introduced it in this blog post which was written for Azure Spring Clean. It allows you to report on monthly costs across multiple Azure Subscriptions and compare those costs to one or more previous months. It also generates some simple charts using another module called PSparklines. The idea is to have a single command you can run once a month (or schedule) to generate insight into how your costs are changing in Azure.

Should you use it? Yes! (if you’re using Azure and want to better understand or report on your costs), although currently it doesn’t work for CSP subscriptions (where billing is done elsewhere). However I am working on a solution to this, as you can usually schedule CSP costs to be exported to a storage account and then AzCostTools will be able to import those exports.

MacNotify

• First published: 02 Oct 2018
• Last updated: 17 Mar 2024
• Download count: 1221

My second least popular module is an apparently little known tool called MacNotify that I published in 2018. This is for MacOS users of PowerShell, made possible since the introduction of PowerShell Core. MacNotify allows you to generate your own custom MacOS alerts (those little pop up boxes that appear in the top right). If you’ve ever heard of BurntToast by Josh King (which has only been downloaded 19 million times), MacNotify is the Mac equivalent. In fact Josh went on to create a cross-platform module called PoshNotify which wraps BurntToast, MacNotify and PSNotifySend (for generating notifications under Linux) so that you can use consistent cmdlet names to generate notifications on any of those operating systems.

Should you use it? Yes! In fact I made some updates to it a few months ago as I was clearing down old issues on my GitHub repos and as part of that I made sure it still works on the current version of MacOS (which it does!) so if you’re running PowerShell on MacOS and want an easy way to surface events MacNotify could be what you’re looking for. Or use PoshNotify and make your script OS notifications platform agnostic.

PSHipChat

• First published: 25 Jan 2017
• Last updated: 09 Sep 2019
• Download count: 1283

I first uploaded PSHipChat to GitHub in March of 2016. HipChat is (was?) Atlassian’s answers to Slack. I think they bundled it free with some of their other products (such as Confluence) as I worked for a few companies where it was used before Slack became more ubiquitous and now (for me at least) Teams. The PSHipChat module provides some PowerShell cmdlets for sending notifications into HipChat from your scripts by invoking its API (spoiler alert: wrapping an API is the theme of a lot of my modules, FYI). This allowed my team at the time to get visibility of when certain automated tasks were running, by just adding a few lines to those scripts to trigger notifications in HipChat.

The original blog for it is here.

Should you use it? I have no idea tbh. I haven’t worked anywhere with HipChat for a number of years, so I’m not sure if it still works. The project is actually now maintained under the AtlassianPS GitHub organisation, but its not getting a lot of love there either.

XKCD

• First published: 19 Jan 2017
• Last updated: 25 Feb 2020
• Download count: 1584

XKCD is a webcomic of romance, sarcasm, math, and language that’s been going since September 2005. The XKCD module just wraps some PowerShell friendly cmdlets around the XKCD API which requires no key/registration and returns JSON. This was mostly for fun, and an excuse to get my head around creating advanced functions with Parameter Sets.

Should you use it? I don’t see why not! It still works. The most useful thing about it is its ability to search for a specific comic by keyword, and it builds a local cache of the comics inside the Module so subsequent searches require no calls to the API. So if you’re looking for a comic on a specific topic, it’s as good as Google:

Find-Xkcd -Query covid | FT

month  num link year news safe_title              transcript alt
-----  --- ---- ---- ---- ----------              ---------- ---
7     2333      2020      COVID Risk Chart                   First prize is a free ticket to the kissing booth.
8     2346      2020      COVID Risk Comfort Zone            I'm like a vampire, except I'm not crossing that threshold even if you invite me.
9     2355      2020      University COVID Model             I admit this is an exaggeration, since I can think of at least three parties I attended while doing my degree, and I'm probably forgetting several more.
12    2395      2020      Covid Precaution Level             It's frustrating to calibrate your precautions when there's only one kind of really definitive feedback you can get, you can only get it once, and when you do it's too late.

Or you can open a random comic by running Get-Xkcd -Random -Open.

Remedy

• First published: 25 May 2017
• Last updated: 10 Sep 2019
• Download count: 1666

My Remedy module is another API wrapper, this time for the ITSM tool BMC Remedy. At the time I wrote this module I was working somewhere that used BMC Remedy and the interface for it back then was pretty clunky. Being able to pull data out of it via PowerShell allowed me to automate a lot of reporting that I needed to generate, as well as hook it up to the next module in this list (SlackBot) to allow my team to query tickets / changes / assets as part of a Slack conversation.

Should you use it? This is another one where I’m not sure. I haven’t worked somewhere with BMC Remedy for a while. If you’re using the version of Remedy I had when I wrote it (which I think was v6) then I suspect it will still work, but I doubt it’s compatible with any subsequent versions.

SlackBot

• First published: 25 Jan 2017
• Last updated: 10 Sep 2019
• Download count: 1674

The purpose of my SlackBot module was to provide a basic demonstration of how to setup a bot for Slack using their Real-time messaging API. This kind of Bot essentially monitors one or more chat channels for messages (as if it were a user) and can then respond to certain messages directed at it. Part of the motivation for sharing this code was because it was really fiddly to get working at the time (and I wanted to save others the pain).

The version in the PowerShell Gallery (and on GitHub) is just a basic framework that you need to extend with whatever functionality you want the Bot to do. At the company I was working for at the time our internal version of it had lots of additional functionality so you could use to query for assets / tickets / changes etc (as mentioned above) and it would then make calls to other systems to retrieve that data.

Should you use it? Maybe. If you’d rather build something up yourself, I think its still a good starting point, although I am also not a big Slack user these days so I don’t know entirely if it’s still compatible with the API. There is a more featured alternative I can recommend though (that I suspect is also more up to date) and easier to customise/extend with additional functionality: PoshBot by Brandon Olin.

CurrencyConverter

• First published: 07 Feb 2024
• Last updated: 18 May 2024
• Download count: 1675

CurrencyConverter is the newest module in this list. It actually began life as part of AzCostTools, where I wanted to be able to convert Azure costs into an alternative currency (as sometimes they’re billed in a currency that is not your own). I realised this was probably useful functionality in its own right and there didn’t seem to be a recent module in the gallery that could do the same. One of the appealing things about my module is that it uses a backend service that has an open API, so there’s no need to register an account / generate a key. This caused some confusion on Reddit when I shared it as people thought I’d just hard coded my own key into the tool but this is not the case.

Should you use it? Yes! It should work fine as long as the API it wraps continues to exist. I also added crypto currency conversion recently at the request of someone who raised an issue on it’s GitHub project. Beware though that for traditional currencies the exchange rates only update daily.

Lumos

• First published: 07 Aug 2019
• Last updated: 07 Dec 2021
• Download count: 4395

The year was 2019. Windows 10 had just got a dark mode feature, as had MacOS, but neither offered the ability to switch between light and dark mode automatically. Hence Lumos was born, and the world was saved. I think both OSes do allow you to configure automatic switching now, so Lumos is kind of redundant. That being said it does allow you to switch Windows to dark mode, but keep apps as light mode (or vice versa) and it also allows you to configure a specific wallpaper for each mode that it can then switch for you at the same time (can Windows do that? Maybe! I haven’t checked). Plus it’s fun to type Invoke-Lumos and feel like (you’re) a Wizard (Harry).

Should you use it? Yes, be magical.

ADAudit

• First published: 07 Jun 2016
• Last updated: 25 Feb 2020
• Download count: 9800

I always thought XKCD was the first module I ever published, but apparently the title belongs to ADAudit. This module actually just built on some excellent work by Irwin Strachan, who was very supportive and continues to be a wonderful and generous member of the PowerShell community. ADAudit is just a set of Pester tests which validate whether an Active Directory forest is in good health. The idea is that (assuming your AD is currently healthy) you export a “gold snapshot” of its current state and then can run the tests routinely to see if anything has changed / is out of order.

If you want to read more about it the original blog is here.

Should you use it? I would say probably not, unless you fancy bringing it up to date. I haven’t updated it for several years and so it’s currently designed for use with Pester v3 (which is pretty old). A good alternative looks to be Testimo by EvotecIT, although I haven’t personally used it, it looks to be more featured and recently maintained.

Watch

• First published: 19 Mar 2018
• Last updated: 08 Mar 2023
• Download count: 14898

Linux has a Watch command and Windows doesn’t which was frankly unacceptable. Watch allows you to run a command (or script block) continuously until the output changes. It’s useful if you’re waiting for something to occur, or want to catch something in the act (like a process starting, or terminating). Ultimately all we’re really doing here is wrapping some code in a while loop and comparing it’s output to the previous iteration. One of the coolest things about Watch-Command is that you can just pipe a string of commands to it. It then gets the full pipeline that proceeded it from $MyInvocation so it can loop those commands and watch for changes. You can also send it a script block, but you don’t have to. This makes it pretty convenient to use.

Get-Service | Select Name,Status | Watch-Command

Should you use it? Yes! It still works as far as I know. I don’t use it in earnest but every now and again I have some need to quickly monitor something for change and I remember it exists. And 14,865 other people apparently did too.

HashCopy

• First published: 06 Aug 2018
• Last updated: 02 Jul 2023
• Download count: 24316

24.3K downloads (you can tell we’re getting serious now, because I’m using “K”). HashCopy came into existence because I needed to know if some files in one folder matched some files in another folder, but I couldn’t trust the modified dates to tell me (long story short: Git). I discovered PowerShell has a Get-FileHash command, which you can use to compute the hash value of a file. So by computing the hash values of the files I wanted to compare, I could quickly determine if they were different.

Should you use it? Yes! I guess I’m not the only person to face this problem based on the downloads, and I don’t think much has changed for Get-FileHash so it should still work. You can read the original blog post here.

Subnet

• First published: 11 Apr 2019
• Last updated: 10 Sep 2019
• Download count: 432312

Now we take a giant leap to the last two modules that do most of the heavy lifting for this two-thumbed millionaire. If I’m being honest here, I suspect much of the success of my Subnet module is because I noticed no one had nabbed the name “Subnet”. It’s possibly an oversight (and a bit of a security flaw) of the PowerShell Gallery that anyone can take any available name.

The Subnet module has 3 commands, Get-Subnet calculates details of a specified subnet (such as it’s subnet mask, IP range and the host addresses), Get-NetworkClass tells you the network class of a specified IP and Test-PrivateIP gives you a true/false result for public/private IPs.

Should you use it? Yes! I personally use it pretty frequently as it’s usually more convenient than using a subnet calculator website. It’s also a great tool if you need to generate subnet details for a list of IP addresses.

Influx

• First published: 31 Dec 2017
• Last updated: 06 May 2023
• Download count: 468667

And finally, with a horrifying 468,659 downloads and counting is Influx. Guess what? I had a need to write some metrics into Influx (which is a time-series database that I heartily recommend, particularly if you pair it with Grafana), using the API directly was a little clunky, and so I did something I’d ~never~ done before and turned to PowerShell. The Influx module makes writing statistics into an Influx database pretty simple. And I guess lots of other people thought so too.

Read the original blog post here.

Should you use it? I mean, I guess?! One regret I have is that I bundled into the module various other commands that I wrote for retrieving values from things like VMWare and 3PAR and then writing those into Influx and I think that kind of bloats the module now, but for all I know 100k people think those things are really useful.

If you’ve made it this far, thanks for indulging me through this walk down memory lane. You’ve perhaps noticed there’s a consistent theme to the modules above, which is that most of them started life as a way to solve some specific problem for me, that I realised could probably also help someone else. None of the modules above are perfect, but I suspect they’ve saved (perhaps a lot) of people (hopefully a lot) of time, and that’s worth a million. Here’s to the next.

Maybe the real treasure was the automation improvements we made along the way.

Originally Azure DevOps featured two sections for building pipeline automation, named Builds and Releases. Several years ago Microsoft renamed the Builds section to just “Pipelines” to better reflect that they can be used to create automation for any purpose (including builds or releases), or if you’re doing CI/CD, a single pipeline that both builds and releases. Pipelines that you create under “Pipelines” can still be built via the UI, or they can be stored as one or more YAML files in source control. In contrast, pipelines built under “Releases” can only be configured via the Azure DevOps UI. These pipelines do feature a version history, but its only visible within Azure DevOps.

Getting started

Getting started with YAML pipelines can be a little intimidating. If you go to Pipelines (underneath Pipelines..) and click the “New Pipeline” button in the top right, Azure DevOps will take you through a sort of setup wizard (because Microsoft 😝). You’ll first need to select where your code is, which can be an Azure DevOps repository, Github repository, bitbucket cloud or several other option (it’s nice that you can use Azure DevOps to build or automate code that lives elsewhere). Once you’ve picked a location, its going to ask you if you have an existing pipeline definition (i.e a YAML file that is already in the repository) or if you want to create a new one. You can choose “starter pipeline” to have a very basic scaffolding at the top, or choose from a number of templates for different build purposes (it will suggest templates based on the kind of code it finds in the repository you selected). But all of this is very build focussed, and there’s nothing overly useful here for someone looking to create a deployment pipeline.

If you’re converting a Classic Build pipeline to YAML, the process is relatively simple as you can export the whole pipeline as YAML via one step. See the official guidance here

Unfortunately there’s not a built-in way to do the same for Classic Release pipelines. This third party tool can apparently do it, but I didn’t personally try it so use at your own risk:

• https://github.com/f2calv/yamlizr

Personally I started from scratch, so I suggest finding a suitable location in whatever repository you want to store your pipeline, and create a new file there called (for example) Deployment.yml (or whatever you want to name it).

Working with YAML pipelines can be fiddly, because YAML requires things be indented correctly. I recommend using Visual Studio Code, and installing the Azure Pipelines Extension as well as an extension to help with formatting YAML, such as Prettier

If you’re new to YAML pipelines I also recommend reading the Azure DevOps pipelines documentation, to familiarise yourself with things such as the key concepts.

The Classic Release pipelines I migrated were separated into stages, with a single stage per environment. We would then create a Release, which would consume the artifacts from several builds that contained the built version of the product being deployed, and manually trigger each environment stage whenever we were ready to deploy. There’s lots of different ways to implement this as YAML, I could create a pipeline per environment, or have pipelines that deploy to multiple environments, but what I decided to do was use Templates, so that the pipeline could be defined once, but have an environment parameter in it to specify which environment was being deployed.

Parameters

To specify parameters, your YAML file needs a parameters section, under which you create one or more parameters by specifying a name, type and default value. If you want the parameter to be a drop-down selection, you can specify a list of accepted values:

parameters:
  - name: Environment
    type: string
    values:
      - "Test"
      - "Staging"
      - "UAT"
      - "Production"
    default: "Test"

When the pipeline is executed, the environment selection needs to make sure that we’re using variables specific to that environment. Under the Classic Release pipeline, if you took the approach of having a stage per environment, you could then have variables scoped to that environment (stage). To achieve the same result for the YAML pipeline, we can create Variable Groups, which we link to the template.

Variable Groups

To create Variable Groups, in Azure DevOps go to Pipelines > Library. You might want to create a variable group called “All environments”, which has default values, or values that apply to all environments. On the Classic Release pipelines the equivalent of this would be variables that were Release scoped. You might also want to create variable groups that apply to more than one of your environments, for values that are the same for all environments of that type, for example: “Non-production environments” and “Production environments”. Finally you want to create a variable group for each specific environment, for example: “Test”, “Staging”, “UAT” etc. Within each of your variable groups, create the various environment specific variables that are required for deployment.

Note that with the Classic Release pipelines, the variables you created on a pipeline would be cloned when you created a Release, and then any changes you made to the variables on that Release would affect it only. With variable groups, the variables are read from the group at time of execution, so bear in mind that any changes you make to the variables that reference your groups will affect old and new deployments alike.

Variables

Having created your variable groups, you can now reference them in the YAML template, via a variables section. The order in which you specify each group is important, if the same named variable is in multiple groups the last one defined will be used. So define them in priority order. For example:

variables:
  - group: "All environments"

  - ${{ if in(parameters.Environment, 'Test','Staging') }}:
      - group: "Non-production environments"

  - ${{ if in(parameters.Environment, 'UAT', 'Prod') }}:
      - group: "Production environments"

  - group: ${{ parameters.Environment }}

In the above I’ve used a conditional expression to define the inclusion of the prod/non-prod groups based on the Environment parameter matching one or more specified names. And then I just use the Environment parameter value itself to include the variable group that I created for the specified environment. The template expression syntax ${{ }} is used to include these bits of logic. These get processed when the pipeline is initialised.

Resources

Because we have separate build pipelines (that generate our deployment artifacts such as infrastructure templates and a compiled release of the software) we next define a resources section. Resources are anything used by a pipeline that live outside the pipeline. This can include other Azure DevOps pipelines as well as builds from other external CI systems.

In my case, I need to consume some other Azure DevOps pipelines, for example one that builds my infrastructure artifacts (such as ARM templates and deployment scripts) and one that builds the product itself. Because these are Azure DevOps pipelines, we consume them by specifying pipelines. The value given for pipeline is the alias I want to refer to them via later in the code, project is the name of the Azure DevOps project they exist in, source is the actual name of the pipeline, and version is the specific build version I want to consume.

resources:
  pipelines:
    - pipeline: "InfraBuild"
      project: "MyProject"
      source: "Build_Our_Infrastructure"
      version: 1.0.123

    - pipeline: "ProductBuild"
      project: "MyProject"
      source: "Build_Our_Product"
      version: 1.0.456

Trigger

The next thing I define in my YAML pipeline is the trigger. By default pipelines trigger whenever there is a commit to any branch in their respective repository, but because this is a deployment pipeline that I want to trigger manually, we need to set trigger to none.

trigger: none

Stages

We’re now ready to start defining the actual tasks the deployment pipeline will perform. You don’t have to use stages as part of this, but by grouping the different associated sets of tasks into stages you can then easily enable or disable them at the point where you run the deployment. For example you might have a pipeline that deploys databases, configuration and then the application/s. It might make sense to split these into stages if you might have a run in the future where you only want to execute a subset of them.

To define stages we need to specify stages: and then under that each stage with a name. Within the stage we can define displayName to have a more descriptive title appear in the UI when the pipeline is run. You use pool to define which of your Azure DevOps deployment pools will execute this task. You can define this at the top level if all of your tasks will run under the same pool, but for my pipeline I wanted some tasks to run via a self hosted agent and some tasks to run via Microsoft-Hosted agents, so it was necessary to define it at the stage level.

Within each Stage you can have one or more jobs. We use a special job type here called deployment to indicate to Azure DevOps that this is a deployment job we’re running. Under deployment you specify environment which then allows you to view your deployments per environment under the Pipelines > Environments section of Azure DevOps. In the example below i’m using the Environments parameter value to populate this via the expression syntax. We then need to specify strategy because this is a deployment job. For my purpose this needs to be runOnce because I’m just deploying to a single environment and I just want each phase of the deployment to run once. You can split your tasks under different phases (predeploy, deploy, routeTraffic and postRouteTraffic) but I personally just have all my tasks under deploy. Finally you use steps: under which you then define each of the tasks to be performed.

Here’s an example first stage, with some tasks for unzipping and deploying a database DACPAC:

stages:
  - stage: DatabaseDeployment
    displayName: "Deploy Database"
    pool: "My Deployment Pool"
    jobs:
      - deployment: DatabaseDeployment
        environment: ${{ parameters.Environment }}
        strategy:
          runOnce:
            deploy:
              steps:
                - download: ProductBuild
                  artifact: Database
                  displayName: Download Database Release

                - task: ExtractFiles@1
                  displayName: "Extract Database zip file"
                  inputs:
                    archiveFilePatterns: 'ProductBuild\Database\Database.zip'
                    destinationFolder: 'ProductBuild\Database'
                    cleanDestinationFolder: false

                - task: AzureKeyVault@1
                  displayName: "Get the Database password from KeyVault"
                  inputs:
                    azureSubscription: "$(azureSubscription)"
                    KeyVaultName: "$(KeyVaultName)"
                    SecretsFilter: DatabasePassword

                - task: SqlAzureDacpacDeployment@1
                  displayName: "Deploy Product Database"
                  inputs:
                    azureSubscription: "$(azureSubscription)"
                    ServerName: "productdb-$(EnvironmentName).database.windows.net"
                    DatabaseName: ProductDatabase
                    SqlUsername: databaseadmin
                    SqlPassword: "$(DatabasePassword)"
                    DacpacFile: "ProductBuild/Database/ProductDatabase.dacpac"