0

I've got a recursive function that loops over an object. It looks for keys with the name subcomponent (which is always an array) and performs an asynchronous function on each of subcomponent's children, the output of which is used to replace the child's data.

In the example below populateSubcomponent() is the async function.

Code:

async function doPopulate(data) {
    Object.keys(data).map((key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promises = data[key].map(subcomponent => populateSubcomponent(subcomponent));
                return Promise.all(promises).then((output) => {
                    data[key] = output;
                    console.log('1');
                });
            }
            doPopulate(data[key]); // Check recursively
            console.log('2');
        }
        console.log('3');
        return data;
    });
}
doPopulate(data);

My expectations are that each of the console.log numbers should fire sequentially, but instead I get 2, 3, then 1. As a result, the recursive functions runs before the async function has completed, therefore never replacing the child as intended; I get the correct result at 1 but it's not passed to 2 or 3.

How do I best incorporate the recursive doPopulate() call with the if statement?

I've looked at the following SO posts:

but I can't relate any of the answers to my own problem, mostly due to the fact that I've got an if statement within my recursive function and I'm not sure how to deal with that in context of the async stuff.

Edit

Thanks to everyone's comments I came up with the following:

async function doPopulate(data) {
    const promisesOuter = Object.keys(data).map(async (key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promisesInner = data[key].map(subcomponent => populateSubcomponent(subcomponent));
                data[key] = await Promise.all(promisesInner);
            }
            await doPopulate(data[key]); // Check recursively
        }
        return data;
    });
    return Promise.all(promisesOuter);
}
return doPopulate(data);

As this all happens within a NodeJS stream (using through2) I also needed to make the stream function async too:

const through = require('through2');

return through.obj(async (file, enc, done) => {
    const data = JSON.parse(file.contents.toString());
    await doPopulate(data);
    file.contents = Buffer.from(JSON.stringify(data));
    return done(null, file);
});
8
  • Use await doPopulate(data[key]). In order to do this you will have to make the callback to .map async as well. In fact you don't need to make doPopulate async; you just need to return the promise. Commented Jul 26, 2018 at 18:26
  • @ExplosionPills Object.keys(data).map(async (key) => {(line 2) and await doPopulate(data[key]) (line 14) don't seem to do anything... Aren't I already returning the promises with return Promise.all(promises).then((output) => {? Commented Jul 26, 2018 at 18:31
  • No, because in your current iteration doPopulate(data[key]) runs before data[key] = output; Commented Jul 26, 2018 at 18:33
  • @KevinB - Yup... That's basically my issue - I'm not sure how to correctly structure my code so that doPopulate(data[key]) waits for the if statement to complete... Commented Jul 26, 2018 at 18:47
  • @GarethJames Why can't it be in the promise's .then AND in an else statement? Commented Jul 26, 2018 at 18:48

2 Answers 2

0

You will have to await every promise sequentially as long as the next promise relies on the previous one. If they can be done concurrently, you just have to aggregate the promises and await them together with Promise.all. By the way you only need the async keyword if you need to use await. Otherwise there's no reason to make the function async.

If you wanted to wait for each promise sequentially you would rewrite it like this:

function doPopulate(data) {
    return Object.keys(data).map(async (key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promises = data[key].map((subcomponent) =>
                    populateSubcomponent(subcomponent)
                );
                data[key] = await Promise.all(promises);
                console.log('1');
            }
            await doPopulate(data[key]); // Check recursively
            console.log('2');
        }
        console.log('3');
        return data;
    });
}
Sign up to request clarification or add additional context in comments.

1 Comment

Cheers for that! Didn't quite work but used data[key] = await Promise.all(promises) to figure out the answer...
0

Turns out I needed to return two sets of promises as @Bergi suggested in the comments:

async function doPopulate(data) {
    const promisesOuter = Object.keys(data).map(async (key) => {
        if (typeof data[key] === 'object') { // We want to do a recursive check on the data so are interested in any objects (arrays) at this point
            if (key === 'subcomponent') { // If we have an object (array) with key `subcomponent` we want to populate each of its children
                const promisesInner = data[key].map(subcomponent => populateSubcomponent(subcomponent));
                data[key] = await Promise.all(promisesInner);
            }
            await doPopulate(data[key]); // Check recursively
        }
        return data;
    });
    return Promise.all(promisesOuter);
}
return doPopulate(data);

As this all happens within a NodeJS stream (using through2) I also needed to make the stream function async too:

return through.obj(async (file, enc, done) => {
    const data = JSON.parse(file.contents.toString());
    await doPopulate(data);
    file.contents = Buffer.from(JSON.stringify(data));
    return done(null, file);
});

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.