1

I am writing some Terraform automation for Azure DevOps and cannot quite figure out how to best do what I want to.

I have a Pull Request pipeline, that on opening a pull request, undertakes security and end-to-end testing (using an ephemeral environment) this all works fine. Once those have completed, that same pipeline generates pipeline artifacts containing the saved plan and binaries used for each environment that has changes in the plan.

I then have a deployment pipeline, that triggers on merge into main. This uses a script to check for the existence of the artifacts (per environment) and if they exist, deploys to environments in order (e.g. dev -> stg -> prd). Again, this is working, but I've realised a flaw in my logic.

Currently the artifacts are retrieved by finding the latest successful build on the PR pipeline and downloading the artifacts. If however, there is more than one pull request open, this logic doesn't work as it may download the artifacts for the second (as yet unmerged) PR.

So my question is this - how can I best achieve what I'm looking to do - generate artifacts on a PR and hand them to some sort of deployment pipeline AFTER the pull request has been closed and merged.

Current approach is as follows:

artifactName="$CONFIGURATION_WORKSPACE-$ENVIRONMENT-plan-and-binaries"

pullRequestResponse=$(curl -s -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" "$SYSTEM_COLLECTIONURI/$SYSTEM_TEAMPROJECT/_apis/git/pullrequests?api-version=7.2-preview.1")

# Get the pipeline ID based on the pipeline name
pipelineResponse=$(curl -s -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" "$SYSTEM_COLLECTIONURI/$SYSTEM_TEAMPROJECT/_apis/pipelines?api-version=7.2-preview.1")
pipelineId=$(echo "$pipelineResponse" | jq -r '.value[] | select(.name == "Terraform PR Pipeline" and .folder == "\\TerraformPipelines\\'"$PROJECT_SHORTNAME"'") | .id')

pipelineRunResponse=$(curl -s -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" "$SYSTEM_COLLECTIONURI/$SYSTEM_TEAMPROJECT/_apis/pipelines/$pipelineId/runs?api-version=7.2-preview.1")
latestRunId=$(echo "$pipelineRunResponse" | jq -r '.value | map(select(.result == "succeeded")) | max_by(.finishedDate) | .id')

artifactResponseCode=$(curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $SYSTEM_ACCESSTOKEN" "$SYSTEM_COLLECTIONURI/$SYSTEM_TEAMPROJECT/_apis/pipelines/$pipelineId/runs/$latestRunId/artifacts?artifactName=$artifactName&api-version=7.2-preview.1")

if [ "$artifactResponseCode" -eq 200 ]; then
    echo "Artifacts exist for latest run ID $latestRunId. Proceeding with download, extract and deploy."
    echo "##vso[task.setvariable variable=artifactsExist]true"
else
    echo "Artifacts do not exist for latest run ID $latestRunId. Skipping subsuquent steps."
    echo "##vso[task.setvariable variable=artifactsExist]false"
fi
Improve this question
3
  • Hi there, Have you got a chance to test the suggestions in the answers below. If either of them can meet your requirement, you may consider accepting it. It will be helpful to other members who has the same issue. Thx. Commented Dec 12, 2023 at 2:09
  • 1
    Hey unfortunately not - work and home has been absolutely manic of late! I hope to get chance to lab this out and test over the Xmas break! I definitely want to and to give credit where due :) Commented Dec 13, 2023 at 8:49
  • Sure and take care. Just as a reminder, the suggestions below both mentioned the reconsideration on what is the best practice to trigger your pipeline. Merry Xmas and Happy 2024! Commented Dec 13, 2023 at 10:54

2 Answers 2

Reset to default
1

Update

Based on the further discussions, I would suggest we trigger the deployment pipeline (TerraformDemoDeployment) by a PR-merged commit (IndividualCI) and in this pipeline, we can configure DownloadPipelineArtifact@2 task to download the latest artifacts published by the merged PR with the filter of specific branch refs/pull/$(prId)/merge.

Prerequisite Considerations/Knowledge

  • DownloadPipelineArtifact@2 supports downloading Latest from sepcific branch;
  • The triggering Build.SourceBranch of a PR validation pipeline (TerraformDemoPR) is refs/pull/$(prId)/merge;
  • By default, the message of the merge commit message begins with Merged PR $(prId):;
  • Therefore, we can split the value of $prId from the predined variable $(Build.SourceVersionMessage) that comes with the PR-merged commit trigger - IndividualCI.

TerraformDemoPR.yml

TerraformDemoPR is the PR validation pipeline that generates pipeline aritfacts of tfplan file(s). Having created the PR with the Id of 55 from dev to main, it is triggered with the code from branch ref/pull/55/merge. enter image description here

pool:
  vmImage: windows-latest

variables:
- group: VG-TerraformAuthSP-Secret

trigger: none

pr:
  branches:
    include:
    - main

stages:
- stage: PR
  jobs:
  - job: PRTerraformRG
    displayName: PR validation - Terrafrom RG
    condition: eq(variables['Build.Reason'], 'PullRequest')
    steps:
    - script: |
        choco install terraform
      displayName: Install Terraform
    - script: |
        env
        echo "1. Start initializing"
        terraform init
        echo "2. Start validating"
        terraform validate
        echo "3. Start Planning"
        terraform plan -input=false -var "RGNAME=rg-tf-demorg-sp-$(System.PullRequest.PullRequestId)-Build$(Build.BuildId)" -out "tfplan-rg-tf-demorg-sp-PR$(System.PullRequest.PullRequestId)-Build$(Build.BuildId)"
      displayName: Terraform validatie and plan
      workingDirectory: $(System.DefaultWorkingDirectory)/RG
      env:
        # ARM_USE_MSI: true
        ARM_CLIENT_ID: $(ARM_CLIENT_ID)
        ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
        ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
        ARM_TENANT_ID: $(ARM_TENANT_ID)
    - task: CopyFiles@2
      inputs:
        SourceFolder: '$(System.DefaultWorkingDirectory)/RG'
        Contents: '*tfplan*'
        TargetFolder: '$(Build.ArtifactStagingDirectory)'
    - task: PublishPipelineArtifact@1
      inputs:
        targetPath: '$(Build.ArtifactStagingDirectory)'
        publishLocation: 'pipeline'

There are several other validation builds of this pipeline publishing their artifacts, which are manually re-queued in the PR or automatically triggered by new commits pushed to the source branch dev of PR 55 and also by some other PRs to main. The latest succeeded build of PR 55 validation is the run of 2597.

enter image description here

TerraformDemoDeployment.yml

As soon as the PR 55 is completed and its code is merged to main with the default merge commit message Merged PR 55: XXXXX, the TerraformDemoDeployment pipeline is automatically triggered by this PR-merged commit. The prId is splited from the default merge commit message and the DownloadPipelineArtifact@2 succeeds to download the pubished artifacts of build 2597, even though there are already other newer artifacts published by the builds of TerraformDemoPR.

trigger:
- main

pool:
  vmImage: windows-latest

jobs:
- deployment: TerraformDeploymentJob
  condition: eq(variables['Build.Reason'], 'IndividualCI')
  variables:
    prId: ${{ split(split(variables['Build.SourceVersionMessage'], ':')[0], 'Merged PR ')[1] }}
  environment: E-TerraformDeployment
  strategy:
    runOnce:
      deploy:
        steps:
        - download: none
        - powershell: |
            Write-Host "Build.SourceVersionMessage is $(Build.SourceVersionMessage)"
            Write-Host "prId is $(prId)"
          displayName: Get prId from Build.SourceVersionMessage
        - task: DownloadPipelineArtifact@2
          inputs:
            buildType: 'specific'
            project: '97d5fd3d-48ec-4d63-b7df-c6f7b49023ef'
            definition: 'TerraformDemoPR'
            buildVersionToDownload: 'latestFromBranch'
            branchName: 'refs/pull/$(prId)/merge'
            targetPath: '$(Pipeline.Workspace)'
          displayName: Download latest artifacts published by TerraformDemoPR from branch refs/pull/$(prId)/merge
        - powershell: tree $(Pipeline.Workspace) /F /A
          displayName: Show the downloaded artifacts

enter image description here

Limitations

  • We have to make sure the merge commit message begins with default merge commit message if using custom messages;
  • Based on my test, the split function to sperate $(Build.SourceVersionMessage) only works in the template expression when the trigger is IndividualCI, as $(Build.SourceVersionMessage) is not available in template when the trigger is BatchedCI or Manual.

Hope the workaround can meet your requirment.

Legacy

There isn't a method out of the box for the pipeline to be triggered by a PR merge commit and retrieve the corresponding artifacts published by the commit's PR validation pipeline.

Since you would expect the deployment pipeline to be triggered by any commit merged onto main branch, why not generate the artifacts via another pipeline with the merged version of code from main instead of via the PR validation pipeline with the intermediate version of code from a PR?

I am not sure why it is required to generate the artifacts by your PR validation pipeline.

If not, I would suggest using a new stage in the same deployment pipeline with CI trigger from main to publish artifacts, which makes more sense. Here is a very simple structure of this workflow. Kindly note that

If no checkout steps are added in a traditional job, the default behavior is as if checkout: self were the first step, and the current repository is checked out.

Similarly, if no download steps are added in a deployment job,

All available artifacts from the current pipeline and from the associated pipeline resources are automatically downloaded in deployment jobs and made available for your deployment.

trigger:
- main

stages:
- stage: build
  jobs:
  - job: BuildJob
    steps:
    - script: echo "Some build steps in front of publish."
    - publish: $(System.DefaultWorkingDirectory)
      artifact: build-$(Build.BuildId)

- stage: deploy
  jobs:
  - deployment: DeploymentJob
    displayName: Deploy to Environment Test
    environment:
      name: E-Test
    strategy:
      runOnce:
        preDeploy:
          steps:
          - download: current

Thus the artifact is generated and published based on each commit on main branch and the subsequent deployment job can pick up the expected artifact.

Besides, if you would like to trigger the deployment pipeline by the published artifact from PR pipeline rather than by the CI trigger, you may also consider using pipeline resource trigger.

Hope the information can help.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Hi, yeah I am coming to the conclusion that this is going to be my only real option to be honest. The reason for the "why" is because ideally I wanted people to review a Terraform plan as part of the pull request process, and then to deploy the saved plan that has been reviewed. But I can't seem to find a way of achieving what i need to when there are multiple pull requests open at the same time.
As far as I understand, there is no terraform apply action during your PR validation. I think the saved plan can be also generated again with the merged code. Therefore, we don't have to use the artifacts published in the PR pipeline, right? Thus, a new pipeline to do so with a CI trigger by the merged code makes sense.
You are correct - there is no apply. However, the plan could be different, which is why I ideally want to take the saved plan from PR then apply that. If we regenerate the plan on main, there could be differences (i.e. if someone changed something outside Terraform). So someone could review Plan X (on PR) and then Plan Y (on main) could be different. A bit like if you were to rebuild a container image rather than create an artifact.
Hi @MikeGuy, Based on our discussions, I came up with a workaround in the update of my answer. The key was to tell the CI pipeline from which PR the merged commit came. I tested with many methods and decided to split the PR ID from the merge commit message. Please check it at your convenience. And if it can resolve your query, please consider this to help other community members with similar requirements. Thx & cheers!
HI Alvin, that is great - thanks for the effort you've put in here! This looks like it will do what I need it to. I will aim to test tomorrow/Wednesday then mark your solution as the answer if so! Thanks again - super helpful comment.
|
0

I suggest to follow this route:

  1. Find a way to make the Pull Request pipeline to mark the build that you want to deploy
  2. Find a way to make the Deployment pipeline retrieve the right artifact marked

How can we achieve this

I can think about 2 routes/tools to use to achieve this.

One is more "professional", with a long term view via using Azure Artifacts Feeds. The cons of this is that you need to setup a proper versioning system.

The other is just enough to solve your problem without having to introduce too many new pieces in your system, that avoids you to pay the price of store on Azure Artifact but you lose on the pros of versioning your stuff. To achieve this you can use Pipeline Build Tags.

Because it looks like the second solution is closer to your situation I'll explain this one.

First you need to set your Pull request pipeline to trigger also when you complete a merge. To do that you can follow this answer.

Also in the same Pull request pipeline you need to add a new step that will tag your build. When I talk about tags Im not talking about Git Tags, but about build tags. They appear inside your run page in the upper left corner, below the run name. They look like this:

You can follow this answer to understand how to tag your build. You can use a tag like "Deployable" or "Ready" or "Release" or whatever you prefer.

Of course you need to create the right condition in your pipeline definition to run this step only when the pipeline is merged.

A good condition could be creating a isRelease variable inside your pipeline definition that looks like this:

- variables:
  "isRelease": and(branchIsMain, buildReasonIsPullRequest)

After you have done this now your Pr pipeline should:

  1. Run also when a PR is merged
  2. Only when a PR is merged it tags the build

Now inside your deployment pipeline you can add as a condition to take the last successful build from PR pipeline that also contains the tag "Release". You can use the DownloadPipelineArtifact@2 task that also exposes the tags parameters that let you filter them.

Improve this answer

2 Comments

Thank you for your input! I shall try this as soon as I can. Much appreciated.
Thanks for this. The build tags is useful to know about and I shall have a play with it, but essentially my challenge here is still the same. Even with the build tags, we are still creating a new artifact on a re-run after merge into main, rather than using the previously created artifact that was reviewed. I think my only option really is to just generate these artifacts after merge into main, and keep the PR purely for code review and security checks.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.