2

Is there a way to write data to an Azure SQL DB from a pandas dataframe in an Azure Machine Learning Service pipeline using the AMLS python SDK? I know I can input SQL data using register_SQL_ds() and I can save output data as a TabularDataset using OutputTabularDatasetConfig(), but how do I write back to an Azure SQL DB?

Ideally I would like to add new rows and update old rows in an Azure SQL table. But just being able to output the data into a temporary SQL table and doing the add/update step afterwards in SQL would work as well.

Improve this question
2
  • By “SQL db” do you mean: Azure SQL DB? Commented Aug 30, 2020 at 1:50
  • Yes, edited my question to make that clear! Commented Aug 30, 2020 at 4:02

1 Answer 1

Reset to default
1

Please follow the below DataTransferStep pointers. Doc: https://learn.microsoft.com/en-us/python/api/azureml-pipeline-steps/azureml.pipeline.steps.data_transfer_step.datatransferstep?view=azure-ml-py Example: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-data-transfer.ipynb

Note: When copying data to an Azure SQL Database, data will be appended to an existing table. We also expect the source file to have a header row and the names should exactly match with column names in destination table.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

Thanks for pointing this out. However I want to export data to an Azure SQL DB that is stored in memory, not in another storage option. I would like to do something similar to what was available for AML Studio: learn.microsoft.com/en-us/azure/machine-learning/….
Can you clarify why you think there isn't feature parity? IMHO using DataTransferStep to output to Azure SQL is equivalent to Azure ML Classic's "Export to Azure SQL Database".
My understanding is that the "export data" feature in AML Studio allowed you to save a pandas dataframe to an Azure SQL DB. DataTransferStep seems to only allow data movement between storage locations (not from a pandas dataframe in memory). Am I understanding this wrong?
I get it now. Yeah I agree it would be convenient to not have to "materialize" (i.e. save it to csv) the pandas df before transferring to Azure SQL.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.