How to run Python script over result generated with U-SQL script in Azure Machine Learning Pipelines?

Question

I want to process large tables stored in an Azure Data Lake Storage (Gen 1), first running on them a U-SQL script, then a Python script, and finally output the result.

Conceptually this is pretty simple:

Run a .usql script to generate intermediate data (two tables, intermediate_1 and intermediate_2) from a large initial_table
Run a Python script over the intermediate data to generate the final result final

What should be the Azure Machine Learning Pipeline steps to do this?

I thought the following plan would work:

Run the .usql query on a adla_compute using an AdlaStep like

int_1 = PipelineData("intermediate_1", datastore=adls_datastore)
int_2 = PipelineData("intermediate_2", datastore=adls_datastore)

adla_step = AdlaStep(script_name='script.usql',
                     source_directory=sample_folder,
                     inputs=[initial_table],
                     outputs=[intermediate_1, intermediate_2],
                     compute_target=adla_compute)

Run a Python step on a compute target aml_compute like

python_step = PythonScriptStep(script_name="process.py",
                               arguments=["--input1", intermediate_1, "--input2", intermediate_2, "--output", final],
                               inputs=[intermediate_1, intermediate_2],
                               outputs=[final],    
                               compute_target=aml_compute, 
                               source_directory=source_directory)

This however fails at the Python step with an error of the kind

StepRun(process.py) Execution Summary

======================================
StepRun(process.py) Status: Failed

Unable to mount data store mydatastore because it does not specify a storage account key.

I don't really understand the error complaining about 'mydatastore', which the name tied to the adls_datastore Azure Data Lake data store reference on which I am running the U-SQL queries against.

Can someone smell if I am doing something really wrong here? Should I move the intermediate data (intermediate_1 and intermediate_2) to a storage account, e.g. with a DataTransferStep, before the PythonScriptStep?

Santhosh Pillai · Accepted Answer · 2019-08-05 20:52:02Z

1

ADLS does not support mount. So, you are right, you will have to use DataTransferStep to move data to blob first.

answered Aug 5, 2019 at 20:52

Santhosh Pillai

1515 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

akshay · Accepted Answer · 2019-08-15 23:32:29Z

Data Lake store is not supported for AML compute. This table lists different computes and their level of support for different datastores: https://learn.microsoft.com/en-us/azure/machine-learning/service/how-to-access-data#compute-and-datastore-matrix

You can use DataTransferStep to copy data from ADLS to blob and then use that blob as input for PythonScriptStep. Sample notebook: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-data-transfer.ipynb


# register blob datastore, example in linked notebook
# blob_datastore = Datastore.register_azure_blob_container(...

int_1_blob = DataReference(
    datastore=blob_datastore,
    data_reference_name="int_1_blob",
    path_on_datastore="int_1")

copy_int_1_to_blob = DataTransferStep(
    name='copy int_1 to blob',
    source_data_reference=int_1,
    destination_data_reference=int_1_blob,
    compute_target=data_factory_compute)

int_2_blob = DataReference(
    datastore=blob_datastore,
    data_reference_name="int_2_blob",
    path_on_datastore="int_2")

copy_int_2_to_blob = DataTransferStep(
    name='copy int_2 to blob',
    source_data_reference=int_2,
    destination_data_reference=int_2_blob,
    compute_target=data_factory_compute)

# update PythonScriptStep to use blob data references
python_step = PythonScriptStep(...
                               arguments=["--input1", int_1_blob, "--input2", int_2_blob, "--output", final],
                               inputs=[int_1_blob, int_2_blob],
                               ...)

Collectives™ on Stack Overflow

How to run Python script over result generated with U-SQL script in Azure Machine Learning Pipelines?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related