This answer uses javascript code in the examples. But it should give you a reference for what is needed in another language: Functions, Vars, and Promises
It is always faster to use IDs for grabbing the contents of a list that is big.
This article describes a way to do this using the nextPage option.
The way to get speed out of it is to break up the total count of items into groups of 5000.
List is 26042 items long with give you six runs
- 1-5000
- 5001-10000
- 10001-15000
- 15001-20000
- 20001-25000
- 25001-26042
Store the smaller number in a var called low and the larger number in high.
Select only the columns or fields you really need to help speed things up, if possible.
Using $top will do two things, force SharePoint to get what ever number is specified (can't go over 5000 regardless of list throttle settings on farm) and remove the nextPage data from being added to your returned object. That will make joining the data together some what easier later on.
What it looks like with the numbers:
/_api/web/lists/getbytitle('ListName')/items?$top=5000&$filter=(Id gt 0) and (Id lt 5001)&$select=columnnames
es6 string interpolation:
`/_api/web/lists/getbytitle('ListName')/items?$top=5000&$filter=(Id gt ${low}) and (Id lt
${high})&$select=columnnames`
OR
a regular string format:
"/_api/web/lists/getbytitle('ListName')/items?$top=5000&$filter=(Id gt "+low+") and (Id lt "+high+")&$select=columnnames
You can build a function that will look at what the last Id (largest Id number) is and then created the batch url strings for you in an array.
Using a Promise.all() you can async call all the urls and let the Promise.all keep them in order. And if for some reason they aren't in order when you join the results, just do a sort on the result by Id.
There are two things to watch out for.
First, a list with the highest Id of 26042 will likely not contain 26,042 items in it. So using the itemCount endpoint of SharePoint will not help you with creating the array of Id batches. If there are missing Id values (for instance, 230 and 564 were deleted, meaning the list is actually only 26040 items long) SharePoint will just ignore them rather than returning null results in their place.
Second, you will need to clean up the results before merging them together to remove the extra 'id' field SharePoint adds. All lists have ID and Id. JSON functions will error out if both are still present when you try to convert from string to an object. A simple find and replace with regex can remove the 'duplicate' heading and value.
