How chunk/parallelize nodejs api requests with await async?

Question

For each id in an array of ids, I want to make an API request to an endpoint somapi.com/todo/{id}. Let's say the API is rate limited to 25 requests per second and a request takes on average 1 second. We'll assume 100000 ids. My first implementation would be something like:

const ids = [...Array(100000).keys()];

const downloadTodos = async (ids) => {
    for (const id of ids) {
      await downloadTodo(id)
    }
}

But that would be slow and far from utilizing the max rate limit. How can I make downloadTodos faster by using parallelization/async-await? Maybe by awaiting N requests in paralell at any given time?

DIY using Promise.all()/allSettled() or any of the many promise queue implementations you can find on NPM. — robertklep
– robertklep, Commented Jul 3, 2022 at 11:24

Andrei · Accepted Answer · 2022-07-03 12:30:04Z

First chunk the array into arrays with 25 ids then using Promise.all or Promise.allSettled wait for the 25 requests to be resolved/rejected and push them in a results array, also make sure at least one second passes between the 25 requests.

const chunkSize = 25;
const waitTimeMs = 1000;

function sleep(ms) {
    return new Promise(resolve => setTimeout(resolve, ms));
}

async function download(ids) {
    const result = [];
    for (let i = 0; i < ids.length; i += chunkSize) {
        // Chunk the ids
        const chunk = ids.slice(i, i + chunkSize);
        const [settled] = await Promise.all([
            // Download a chunk 
            Promise.allSettled(chunk.map(async id => {
                // return downloadTodo(id);
                return Promise.resolve(id);
            })),
            // Make sure at least 1 second passes before the next chunk is downloaded
            sleep(waitTimeMs) 
        ]);
        // Add the chunk to the result
        result.push(...settled.map(s => s.value));
    }
    return result;
}

download([...Array(100).keys()]).then(r => console.log(r));

Collectives™ on Stack Overflow

How chunk/parallelize nodejs api requests with await async?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related