How to handle cross-repository script paths in Azure DevOps YAML pipelines?

Question

I'm working on a multi-repo Azure DevOps setup involving two repositories (Repo1 and Repo2) in different projects, and I'm encountering issues with consistent path references when calling a PowerShell script from a shared YAML pipeline template.

The Setup:

Repo1 (Project1) contains:
- A pipeline, pipeline1.yaml, which calls a YAML template, pipeline_template.yaml.
- The template pipeline_template.yaml then calls a PowerShell script, script.ps1, located in helper_scripts/script.ps1.
Repo2 (Project2) contains:
- A pipeline, pipeline2.yaml, which calls the same pipeline_template.yaml from Repo1.

However, when I run pipeline2.yaml from Repo2, I get a "file not found" error for script.ps1 unless I use the path Repo1/helper_scripts/script.ps1 instead of helper_scripts/script.ps1.

Is there a way to set up a consistent or dynamic file path in pipeline_template.yaml that allows script.ps1 to be referenced correctly when called from both Repo1 and Repo2? I'd like to avoid hardcoding paths for each repository and keep the reference universal if possible. Any advice on how to structure the file paths or configure the pipeline templates to achieve this?

Thank you for any guidance!

pipeline_template.yaml:

- name: parameter1
  type: string
- name: parameter2
  type: string

jobs:
  - job: Job1
    steps:
      - checkout: self
      - task: AzureCLI@2
        inputs:
          azureSubscription: '$(GLOBAL_SERVICE_CONNECTION)'
          scriptType: 'pscore'
          scriptLocation: 'scriptPath'
          scriptPath: 'helper_scripts/script.ps1'
          arguments: '-WebhookURL "${{ parameters.parameter1 }}" -KeyVaultsToCheck "${{ parameters.parameter2 }}"'
        env:
          GLOBAL_SUBSCRIPTION: $(GLOBAL_SUBSCRIPTION)

Hi @Sumit Bhardwaj, you can add checkout: git://project1/repo1 with path repo1 in your pipeline_template.yaml and change scriptPath to $(Agent.BuildDirectory)/repo1/helper_scripts/script.ps1. This is not a hard-coded path, but a new path repo1/ is created in each pipeline that calls this template. In this case, no matter which repo or pipeline this template is used in, it will work as expected. Let me know if you have any other question. — Ziyang Liu-MSFT
– Ziyang Liu-MSFT, Commented Nov 22, 2024 at 7:01

Ziyang Liu-MSFT · Accepted Answer · 2024-11-18 09:57:04Z

In your pipeline_template.yaml, you only add - checkout: self. When you run pipeline2, only repo2 will be checked out, so that the pipeline can't find the PS script in the repo1.

According to Check out multiple repositories in your pipeline,

if a path is specified for a checkout step, that path is used, relative to (Agent.BuildDirectory).

To resolve the above issue, you can checkout repo1 using - checkout: git://project1/repo1 with path and use $(Agent.BuildDirectory)/repo1/helper_scripts/script.ps1 as the path of scripts.

Refer to the example below. (I use PowerShell task for example, it's the same for AzureCLI@2 task.)

pipeline_template.yaml:

jobs:
  - job: Job1
    steps:
    - checkout: self
    - checkout: git://project1/repo1
      path: repo1
    - task: PowerShell@2
      inputs:
        targetType: 'inline'
        script: 'tree /f /a'
        workingDirectory: '$(Agent.BuildDirectory)'
    - task: PowerShell@2
      inputs:
        filePath: '$(Agent.BuildDirectory)/repo1/helper_scripts/script.ps1'

Whether to add - checkout: self depends on whether pipeline2 needs the content in repo2. If not, you can remove this line.

Result:

Pipeline1:
Pipeine2:

wenbo - Finding Job · Accepted Answer · 2024-11-07 03:36:36Z

Way1:

Checkout multiple repos with path, You can use path in checkout, which will download source codes related to $(Pipeline.Workspace), For Linux, locations looks like /home/vsts/work/1/repo1, /home/vsts/work/1/repo2 in your case. Reference docs

So you can define your template like below:

- name: parameter1
  type: string
- name: parameter2
  type: string

jobs:
  - job: Job1
    steps:
      - checkout: self
        path: repo2
      - checkout: repo1
        path: repo1
      - task: AzureCLI@2
        inputs:
          azureSubscription: '$(GLOBAL_SERVICE_CONNECTION)'
          scriptType: 'pscore'
          scriptLocation: 'scriptPath'
          scriptPath: '$(Pipeline.Workspace)/repo1/helper_scripts/script.ps1'
          arguments: '-WebhookURL "${{ parameters.parameter1 }}" -KeyVaultsToCheck "${{ parameters.parameter2 }}"'
        env:
          GLOBAL_SUBSCRIPTION: $(GLOBAL_SUBSCRIPTION)

Way2:

Checkout multiple repos without path, which related to the path $(Build.SourcesDirectory), in Linux equals to /home/vsts/work/1/s. It will download the source code of repos to the path /home/vsts/work/1/s/repo1, /home/vsts/work/1/s/repo2 in your senario.

- name: parameter1
  type: string
- name: parameter2
  type: string

jobs:
  - job: Job1
    steps:
      - checkout: self
      - checkout: repo1
      - task: AzureCLI@2
        inputs:
          azureSubscription: '$(GLOBAL_SERVICE_CONNECTION)'
          scriptType: 'pscore'
          scriptLocation: 'scriptPath'
          scriptPath: '$(Build.SourcesDirectory)/repo1/helper_scripts/script.ps1'
          arguments: '-WebhookURL "${{ parameters.parameter1 }}" -KeyVaultsToCheck "${{ parameters.parameter2 }}"'
        env:
          GLOBAL_SUBSCRIPTION: $(GLOBAL_SUBSCRIPTION)

Shamrai Aleksander · Accepted Answer · 2024-11-06 16:55:44Z

checkout: self downloads only the current repo (Repo2). You may Check out multiple repositories in your pipeline:

- name: parameter1
  type: string
- name: parameter2
  type: string

jobs:
  - job: Job1
    steps:
      - checkout: Repo1
      - checkout: self
      - task: AzureCLI@2
        inputs:
          azureSubscription: '$(GLOBAL_SERVICE_CONNECTION)'
          scriptType: 'pscore'
          scriptLocation: 'scriptPath'
          scriptPath: '$(System.DefaultWorkingDirectory)/Repo1/helper_scripts/script.ps1'
          arguments: '-WebhookURL "${{ parameters.parameter1 }}" -KeyVaultsToCheck "${{ parameters.parameter2 }}"'
        env:
          GLOBAL_SUBSCRIPTION: $(GLOBAL_SUBSCRIPTION)

In this case, you may move your template and referenced scripts to a separate repo to use one approach for Repo1 and Repo2.

      - checkout: TemplateRepo
      - checkout: self
      ......
          scriptPath: '$(System.DefaultWorkingDirectory)/TemplateRepo/helper_scripts/script.ps1'

AhmadYo · Accepted Answer · 2024-11-08 14:50:28Z

-1

Is this solution not good for you to keep helper scripts like helper_scripts/script.ps1 in a separate repository, e.g. pipelinesCommonRepo, and every pipeline in any repo in any project refer to it?
In this case pipelines remain in their repos and any pipeline needs to checkout pipelinesCommonRepo besides its repo. (This solution is near to @shamrai-aleksander but it differs in remaining pipeline definitions in their repo).

helper_scripts in pipelinesCommonRepo

pipeline1.yaml in Repo1 (Project1)

- checkout: self   
- checkout: pipelinesCommonRepo

pipeline2.yaml in Repo2 (Project2)

- checkout: self
- checkout: pipelinesCommonRepo

edited Nov 8, 2024 at 14:50

answered Nov 7, 2024 at 14:53

AhmadYo

4766 silver badges9 bronze badges

Collectives™ on Stack Overflow

How to handle cross-repository script paths in Azure DevOps YAML pipelines?

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related