How to refrence path from python script into yaml file

Question

I have a folder called data which is created from the following code

import shutil
import os

folderName = "..Data"
current_directory = os.getcwd()
final_directory = os.path.join(current_directory,folderName)
if os.path.exists(final_directory):
    shutil.rmtree(final_directory)
if not os.path.exists(final_directory):
    os.makedirs(final_directory)

I want to get the path of this data folder and refrence it in a YAML file. I dont want to manually state the path as it will be run on azure using azure pipelines. is there a way i can get this path from a python script into a YAML file? for reference https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-register-data-assets?tabs=CLI trying to create a URI folder

viggnah · Accepted Answer · 2022-07-28 11:19:08Z

0

If you have the option, you could use the python SDK to create a uri_folder data asset.

You can create a data asset in Azure Machine Learning using [the following] Python Code

Here is the sample provided in the docs, slightly tweaked for your case:

from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes
import shutil
import os

folderName = "..Data"
current_directory = os.getcwd()
final_directory = os.path.join(current_directory,folderName)
if os.path.exists(final_directory):
    shutil.rmtree(final_directory)
if not os.path.exists(final_directory):
    os.makedirs(final_directory)

# Supported paths include:
# local: './<path>'
# blob:  'https://<account_name>.blob.core.windows.net/<container_name>/<path>'
# ADLS gen2: 'abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>/'
# Datastore: 'azureml://datastores/<data_store_name>/paths/<path>'

my_data = Data(
    path=final_directory,
    type=AssetTypes.URI_FOLDER,
    description="<description>",
    name="<name>",
    version='<version>'
)

ml_client.data.create_or_update(my_data)

Update

If yaml is the only option, note that it is possible to embed python code into a yaml file like so:

$schema: https://azuremlschemas.azureedge.net/latest/data.schema.json

# Supported paths include:
# local: ./<path>
# blob:  https://<account_name>.blob.core.windows.net/<container_name>/<path>
# ADLS gen2: abfss://<file_system>@<account_name>.dfs.core.windows.net/<path>/
# Datastore: azureml://datastores/<data_store_name>/paths/<path>
type: uri_folder
name: <name_of_data>
description: <description goes here>
path: |
    import os
    folderName = "..Data"
    current_directory = os.getcwd()
    final_directory = os.path.join(current_directory,folderName)
    print(final_directory)

edited Jul 28, 2022 at 11:19

answered Jul 28, 2022 at 11:07

viggnah

1,9171 gold badge6 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user14736700 Over a year ago

if possible want to avoid using python SDK for this in my use case would be better if I can use azure cli for this. using azure pipelines

viggnah Over a year ago

@muhammad, I have updated the answer, but please test and see.

Collectives™ on Stack Overflow

How to refrence path from python script into yaml file

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related