1

I'm fetching data from an API source, which with each pagination is giving back some meta data needed to control the data flow. Most importantly it gives me the next page URL (captured in the "pagination rules") and the [end_of_stream] value.

However I am failing to understand how I can capture this output value and encapsulate the Copy Activity in a, for example, While-loop (While [end_of_stream] = 0, continue fetching data).

Could someone please assist? Do I need to use the Web Activity instead, as this allows me to view the output of the activity and use those values in variables. The Copy Activity only seems to have the option of assigning values to columns in the sink. And the output of said activity is only showing meta-data not usable for me.

Data Factory Copy Activity

Improve this question

1 Answer 1

Reset to default
1

I'll answer my own question, in case someone else is having the same headache.

What I did not realize is the built-in functionality of the Copy Activity, with the Pagination Rules (reference).

It contains the option of adding an end condition, which can be picked up by the Copy Activity with each iteration. The iteration is also picking up the next page, via the next page URL found in each response from the API (in my case, the Zendesk API).

In my JSON, it looks like:

"EndCondition:$.end_of_stream": "Const:true"

What I would like to do is save the last picked up cursor as an variable, but I guess the best way to do this is by adding this as a column to my source on each row.

Pagination Rules for REST API with end condition:

Pagination Rules for REST API with end condition

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.