1

I am using Azure Data Factory to create the Data pipelines and store the output as JSON files in the Azure blob storage account.

The issue is I am using the Web activity with rest API for retrieving the data and parsing the data along with authorization token to copy activity.

Now I want to filter the output of web activity by removing or skipping the data from web activity before parsing it to the copy activity. So I can save the data by removing or skipping data in the JSON files as I want.

Is it possible to filter the output of web activity before parsing to the copy activity?

Improve this question
2
  • Hi, can you share the sample source and expected output? Commented Jun 21, 2022 at 13:33
  • Actually, the source data is coming from a power bi table using Rest API in JSON format and just need to skip a few columns of the table which is in JSON format of web activity output using filter and parse the output to the copy activity and store the data in JSON file in storage account. Commented Jun 21, 2022 at 14:33

1 Answer 1

Reset to default
2

Here is a simple demo for your usecase. First , im using a jsonPlaceHolder data check this API:

https://jsonplaceholder.typicode.com/users

second , i tried to do the following in Azure Data Factory:

  1. Created a webActivity to call the api(Its a simple GET request - please check the response).
  2. created a set variable activity to save and parse data to be used later.
  3. filtered data according to my condition

Simple ADF pipeline

WebActivity :

Under settings, added the jsonPlaceHolder url and GET method, in headers : Content-Type : application/json.

SetVariable:

Under Variables , Name : data , Value : @json(activity('TestWebActivity').output.Response)

Filter Activity:

Under Settings , items : @variables('data') , condition : @startswith(item().username,'B')

Explanation:

here i set the variable name in set activity to "data", so now in variables you can use data anywhere in your pipeline, in Filter activity , i set the data array in "items" and basically each json in the array is reffered as "item()" , so in order to filter the json array based on a value , in condition you can select the key : items().keyName ..

in order to save your data to Blob storage , add a Copy activity after the filter and link the blob storage as a sink..

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Hi, I just need to know to filter the entire username column from JSON using filter and save the remaining columns as JSON files.
@AzureCloudEnthusiast ok , so you want to remove a key from a Json file. In order to do it , you need to create a Derived Column activity In this activity , you can map the columns that you want ,if you dont map a column it will be deleted from the Json. Derived Column -> Settings -> Columns please check this link : learn.microsoft.com/en-us/azure/data-factory/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.