Rest API call from copy activity

Question

Hi i am processing a set of ~50K records from a pipe delimeted flat kn azure data factory and need to invoke a rest API call for each input record. So, I am using a foreach loop to access each record and inside the loop, I am using a copy activity to invoke a rest API call. My question is, can I invoke the rest API call in bulk for all the records at once, as the foreach loop is slowing the pipeline execution. I want to remove the foreach loop and also process the API json response and store it in azure sql database.

Thanks

Pratik Somaiya · Accepted Answer · 2022-03-04 04:49:28Z

1

You will have to check the Pagination properties so that you can decide how much payload you need to return from source API:

https://learn.microsoft.com/en-us/azure/data-factory/connector-rest?tabs=data-factory#pagination-support

Also, if you need to store the API JSON response in Azure SQL, then you can do so with many built in functions like JSON_PATH

More details can be found in this link:

https://learn.microsoft.com/en-us/azure/azure-sql/database/json-features

answered Mar 4, 2022 at 4:49

Pratik Somaiya

7436 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Rest API call from copy activity

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related