How to deploy a folder of notebooks to Databricks from Azure Devops

Question

I want to deploy databricks notebook using azure devops & terraform.

I get error below:

Error: cannot read notebook: default auth: azure-cli: cannot get access token: ERROR: Please run 'az login' to setup account.

my providers.tf look like this:

terraform {
  required_version = ">1.0.0"

  required_providers {

    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.3.0"
    }

    azuread = {
      source  = "hashicorp/azuread"
      version = ">=2.22.0"
    }

    databricks = {
      source  = "databricks/databricks"
      version = ">=1.28.1"
    }
  }
  backend "azurerm" {
    # "rg-ne-dev-department-project"
    resource_group_name  = "devops"
    storage_account_name = "sadevop"
    container_name       = "tf-plans"
    key                  = "notebooks.state.tfstate"
  }

}

provider "azurerm" {
  features {  }
  use_msi = true
  tenant_id = "xxx"
  client_id       = "xxx"
  client_secret   = "xxx"
  subscription_id = "xxx"

}

provider "databricks" {
  alias = "etl_workspace_provider"
  host  = data.azurerm_databricks_workspace.etl_workspace.workspace_url
}

everything works ok if I login from vs code on laptop but fails on azure devops pipeline.

YAML Pipeline:

trigger:
  - main

pool:
  vmImage: ubuntu-latest

parameters:
 
  - name: Action
    displayName: Action
    type: string
    default: 'Plan'
    values:
    - Plan
    - Apply
    - Destroy
    
variables:
  - name: backendServiceArm
    value: "sp-devops-connection"
  - name: backendAzureRmResourceGroupName
    value: "devops"
  - name: backendAzureRmStorageAccountName
    value: "sadevop"
  - name: backendAzureRmContainerName
    value: "tf-plans"
  - name: backendAzureRmKey
    value: "notebooks.state.tfstate"
  - name: action
    value: ${{ parameters.Action }}



steps:
  - script: |
      echo "(System.DefaultWorkingDirectory): $(System.DefaultWorkingDirectory)"
      echo "(build.ArtifactStagingDirectory) - The directory to which all artifacts are copied before being published.: $(build.ArtifactStagingDirectory)"
      echo "(build.BinariesDirectory) The directory to which all binaries are copied during the build process.: $(build.BinariesDirectory)"
      echo "(build.DefinitionName): $(build.DefinitionName)"
      echo "(build.BuildId): $(build.BuildId)"
      echo "(build.BuildNumber): $(build.BuildNumber)"
      echo "(build.Repository.Name): $(build.Repository.Name)"
      echo "(System.DefaultWorkingDirectory): $(System.DefaultWorkingDirectory)"
      echo "(System.StageDisplayName): $(System.StageDisplayName)"
      echo "(System.JobDisplayName): $(System.JobDisplayName)"
    displayName: "Variable Names"

  - script: |
      echo "Cleaning up build artifacts"
      rm -rf $(System.DefaultWorkingDirectory)/dist
    displayName: "Clean up build artifacts"

  - task: TerraformInstaller@1 # 1st/ install terraform
    displayName: "Install Terraform" # install always need to be installed at every stage
    inputs:
      terraformVersion: "latest"


  - task: TerraformTaskV4@4
    displayName: Initialize Terraform
    inputs:
      provider: "azurerm"
      command: "init"
      backendServiceArm: '${{ variables.backendServiceArm }}'
      backendAzureRmResourceGroupName: '${{ variables.backendAzureRmResourceGroupName }}'
      backendAzureRmStorageAccountName: '${{ variables.backendAzureRmStorageAccountName }}'
      backendAzureRmContainerName: '${{ variables.backendAzureRmContainerName }}'
      backendAzureRmKey: '${{ variables.backendAzureRmKey }}'
      workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'



  - task: AzureCLI@2
    displayName: 'plan '
    inputs:
      azureSubscription: '${{ variables.backendServiceArm }}'
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: 'terraform plan'



  - task: AzureCLI@2
    displayName: 'Azure CLI '
    inputs:
      azureSubscription: '${{ variables.backendServiceArm }}'
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: |
        echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId" 

        echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$servicePrincipalKey"

        echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId
        "
  - task: AzureCLI@2
    displayName: 'CLI Plan '
    inputs:
      azureSubscription: '${{ variables.backendServiceArm }}'
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: terraform plan -out notebook.state.plan

Currently, pipeline fails at displayName: 'plan '

Error: Error: No configuration files │ │ Plan requires configuration to be present. Planning without a configuration │ would mark everything for destruction, which is normally not what is │ desired. If you would like to destroy everything, run plan with the │ -destroy option. Otherwise, create a Terraform configuration file (.tf │ file) and try again.

Service connection is the owner of the Azure subscription.

Kevin Lu · Accepted Answer · 2024-02-21 01:32:58Z

1

Error: cannot read notebook: default auth: azure-cli: cannot get access token: ERROR: Please run 'az login' to setup account.

Based on the error message, it seems that the Terraform is not able to read the azure credentials.

To solve this issue, you can refer to the following two methods:

Method1: You can directly use the Azure CLI task to run the terraform script.

When you use the Azure CLI task, it will automatically execute the az login command based on your service connection. You don't need to add extra az login command.

For example:

steps:
- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: ARM service connection
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: 'terraform apply'

Method2: If you need to run the terraform script in separate tasks. You can use Azure CLI task to export the login information and then use bash/powershell task to execute the az login command. Then the credentials will be pass to next tasks.

For example:

steps:

- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: ARM service connection
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
     echo "##vso[task.setvariable variable=ARM_CLIENT_ID]$servicePrincipalId" 

     echo "##vso[task.setvariable variable=ARM_CLIENT_SECRET]$servicePrincipalKey"

     echo "##vso[task.setvariable variable=ARM_TENANT_ID]$tenantId"
    addSpnToEnvironment: true

- bash: |
   az login --service-principal --username $(ARM_CLIENT_ID) --password $(ARM_CLIENT_SECRET)  --tenant $(ARM_TENANT_ID)'

  displayName: 'Bash Script'

- task: terraform@x
   xxx

Note: We need to set addSpnToEnvironment: true in Azure CLI

For more details, you can refer to this ticket: Terraform doesnt realise Im already connected to Azure?

Update:

Error: No configuration files

When you use Azure CLI to run terraform plan command, you need to make sure that the working directory contains the required configuration files.

  - task: AzureCLI@2
    displayName: 'plan '
    inputs:
      azureSubscription: '${{ variables.backendServiceArm }}'
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: 'terraform plan'
      workingDirectory: '$(build.sourcesdirectory)/terraformfolder'

edited Feb 21, 2024 at 1:32

answered Feb 19, 2024 at 1:31

Kevin Lu

36.9k4 gold badges36 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

BI Dude Over a year ago

Thank you @kevin-lu-msft, both suggestion give me the following: Error: No configuration files

BI Dude Over a year ago

Generating script. ========================== Starting Command Output =========================== /usr/bin/bash /home/vsts/work/_temp/f152ef2d-7e23-4c15-959a-92531122c0fc.sh ERROR: argument --password/-p: expected one argument

BI Dude Over a year ago

Service connection is the owner of the Azure subscription.

Kevin Lu Over a year ago

@BIDude Refer to the updated answer, for the issue about no configuration files, you need to define the working folder for the Azure ClI task to access the correct path of Configuration file.

Kevin Lu Over a year ago

If you want to use Azure CLI to output the related authentication information, you need to set addSpnToEnvironment: true. Please check the sample in the answer. If you don't set the option, it will not output the SP information for the az login in the next task

Collectives™ on Stack Overflow

How to deploy a folder of notebooks to Databricks from Azure Devops

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related