Databricks command not found in azure devops pipeline

Question

I am trying to copy a file to Azure Databricks DBFS through Azure Devops pipeline. The following is a snippet from the yml file I am using:

stages:
- stage: MYBuild
  displayName: "My Build"
  jobs:
    - job: BuildwhlAndRunPytest
      pool:
        vmImage: 'ubuntu-16.04'

      steps:
      - task: UsePythonVersion@0
        displayName: 'Use Python 3.7'
        inputs:
          versionSpec: '3.7'
          addToPath: true
          architecture: 'x64'

      - script: |
          pip install pytest requests setuptools wheel pytest-cov
          pip install -U databricks-connect==7.3.*
        displayName: 'Load Python Dependencies'

      - checkout: self
        persistCredentials: true
        clean: true

      - script: |
          echo "y
          $(databricks-host)
          $(databricks-token)
          $(databricks-cluster)
          $(databricks-org-id)
          8787" | databricks-connect configure
          databricks-connect test
        env:
          databricks-token: $(databricks-token)
        displayName: 'Configure DBConnect'

      - script: |
          databricks fs cp test-proj/pyspark-lib/configs/config.ini dbfs:/configs/test-proj/config.ini

I get the following error at the stage where I am invoking the databricks fs cp command:

/home/vsts/work/_temp/2278f7d5-1d96-4c4e-a501-77c07419773b.sh: line 7: databricks: command not found

However, when I run databricks-connect test, it is able to execute the command successfully. Kindly help if I am missing some steps somewhere.

Alex Ott · Accepted Answer · 2021-05-11 06:13:52Z

3

The databricks command is located in the databricks-cli package, not in the databricks-connect, so you need to change your pip install command.

Also, for databricks command you can just set the environment variables DATABRICKS_HOST and DATABRICKS_TOKEN and it will work, like this:

- script: |
    pip install pytest requests setuptools wheel
    pip install -U databricks-cli
  displayName: 'Load Python Dependencies'

- script: |
    databricks fs cp ... dbfs:/...
  env:
    DATABRICKS_HOST: $(DATABRICKS_HOST)
    DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)
  displayName: 'Copy artifacts'

P.S. Here is an example on how to do CI/CD on Databricks + notebooks. You could be also interested in cicd-templates project.

answered May 11, 2021 at 6:13

Alex Ott

88.1k10 gold badges110 silver badges157 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Databricks command not found in azure devops pipeline

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related