Best way to iterate and make an async call during each iteration

Question

If you have to loop and make a bunch of calls to a repository or gateway in my case, how do I do that asynchronously meaning not wrapping my async calls inside a synchronous for loop?

For example, what would be a better approach (restructuring this code) to loop through a set of ids, and make the call to find() below like I'm trying to do?

The goal: I want to take an array of ids, iterate them, and during each iteration, use the id to call find() on my gateway to go get the object for that id, then stuff it into a final array in which I'll return when all said and done.

What I'm using:

q (for promises)
co-pg (to hit the database)

someModule.js

var _gateway = require('./database/someGateway');

          var cars = [];
          var car;

             for (var i = 0; i < results.docs.length; i++){

                 var carId = results.docs[i].carId;

                 _gateway.find(carId)
                 .then(function(data){
                         console.log('data[0]: ' + data[0].id);
                         cars.push(data[0]);
                     })
                 .done();
             }

console.log("cars: " + cars.length); // length here is 0 because my asyn calls weren't done yet
             result(cars);

someGateway.js

'use strict';
var Q = require('q');

var _carModel = require('../../models/car');

module.exports = {
    models: {
        car: _carModel
    },
    find: _find
};

function _find(carId)
{
    return _carModel.find(carId);
};

carModel.js

'use strict';

var Q = require('q');
var pg = require('co-pg')(require('pg'));
var config = require('../../models/database-config');

var car = module.exports = {};

car.find = Q.async(function *(id)
{
    var query = 'SELECT id, title, description FROM car WHERE id = ' + id;

    var connectionResults = yield pg.connectPromise(config.connection);

    var client = connectionResults[0];
    var done = connectionResults[1];

    var result = yield client.queryPromise(query);
    done();

    console.log("result.rows[0].id: " + result.rows[0].id);
    return result.rows;
});

so I need help understanding how to refactor my code in someModule.js to get that working properly, so that I make a call to find() for each id, stuff each found car into the array, then return the array. The carModel code is async. It goes out to a physical database to perform the actual query lookup.

UPDATE #1

Ok after a couple more hours of trying all sorts of sh** (q.all(), and a ton of other combinations of callback code, etc.) here's what I have at this point:

someModule.js

var _data;
var Q = require('q');
var _solrClient = require('../models/solr/query');
var _solrEndpoint = "q=_text&indent=true&rows=10";
var _postgreSQLGateway = require('./database/postgreSQLGateway');

module.exports = {
    data: function(data){
        _data = data;
    },
    find: function (text, result){

        if(!searchText){
            result(null);
        };

         _solrClient.query(endpoint, function(results){

             var carIds = [];
             var cars = [];
             var car;

             for (var i = 0; i < results.docs.length; i++){
                 carIds.push(results.docs[i].carId);
             }

             for (var i = 0; i < carIds.length; i++) {

                 var car = _postgreSQLGateway.find(carIds[i], function(o){
                     console.log("i: " + i);
                 });
             };
        });
    }
};

someGateway.js

'use strict';
var Q = require('q');

var _carModel = require('../../models/postgreSQL/car');

module.exports = {
    models: {
        car: _carModel
    },
    find: _find
};

function _find(carId, foundCar)
{
    console.log("CALL MADE");

    _carModel.find(carId)
        .then(function(car){
            console.log("car: " + car[0].id);

            foundCar(car);
        });
};

carModel.js

[same code, has not changed]

Of course I noticed that the for loop fires off all my function calls asyncronously and so when I console.write the i, it's 10 because the for loop is done but then as we know, the rest of the console.logs happen later after the callbacks are done.

So I still can't get this working right...

Also when I was playing around I started down this path but it ended at a brick wall:

var find = Q.async(function(carIds, cars)
{
    var tasks = [];
    var foundCars = [];

    for (var i = 0; i < carIds.length; i++) {
        tasks.push(_postgreSQLGateway.find(carIds[' + i + ']));
    };

    Q.all([tasks.join()]).done(function (values) {
        for (var i = 0; i < values.length; i++) {
            console.log("VALUES: " + values[0]);
            foundCars.push(values[0]);
        }
        cars(foundCars);
    });
});

I ended up with [object promise] every time for values[i] instead of a car for value[i]

Perhaps bring the _gateway.find(carId) outside of the for loop and store it. var r = _gateway.find(results.docs[0].carId); Then in the loop, say r = r.then(_gateway.find.bind(results.docs[i])).then(...), and after the loop, call r.done(...). Disclosure I haven't used the Q library before — azz
– azz, Commented Jul 31, 2015 at 2:51
Just looking at the Q docs, it looks like you could use Q.all([ ... ]), where the array consists of Q promises. — azz
– azz, Commented Jul 31, 2015 at 2:58
Learning how to do good, reliable, maintainable async development is THE key skill to learn in node.js development. Once you learn it and learn how to use tools like promises and other libraries to help you manage it and understand the design choices to make and the implementation tools to use, it's not hard. But, there's a learning curve to get yourself up to speed. — jfriend00
– jfriend00, Commented Jul 31, 2015 at 5:22

jfriend00 · Accepted Answer · 2015-07-31 05:36:29Z

3

I don't know the Q promises library, but here's a solution using generic Promises built into node.js. This runs all the requests in parallel and then when all results have been collected, it runs the final .then() handler with all the results:

var _gateway = require('./database/someGateway');

var promises = [];
for (var i = 0; i < results.docs.length; i++) {
    promises.push(_gateway.find(results.docs[i].carId).then(function (data) {
        console.log('data[0]: ' + data[0].id);
        return data[0];
    }));
}
Promise.all(promises).then(function(cars) {
    // cars will be an array of results in order
    console.log("cars: " + cars.length);
    result(cars);
});

Individual promise libraries (like the one I know Bluebird) have features built in that lets you do this kind of activity in even less code, but I've intentionally kept this answer to just using standard promise features.

answered Jul 31, 2015 at 5:36

jfriend00

711k104 gold badges1.1k silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

PositiveGuy Over a year ago

thanks I'm sure this works also...you have the same idea

greim · Accepted Answer · 2015-07-31 05:54:26Z

2

This is potentially really easy with the vanilla Promise API from es6 (and replicated by Bluebird and other libs). First map the IDs to an array of promises:

var promises = results.docs.map(function(doc) {
  return _gateway.find(doc.carId);
});

Then create a promise for the aggregate result:

var allDone = Promise.all(promises);

Then inside the done() callback of the aggregate promise, you'll have a final array of results, in the same length and order as the carId array:

allDone.then(function(results) {
  // do something with "results"
});

edited Jul 31, 2015 at 5:54

answered Jul 31, 2015 at 5:33

greim

9,4676 gold badges36 silver badges35 bronze badges

3 Comments

PositiveGuy Over a year ago

for the map, it's just a plain js map() which says take each object in docs and return the result for the function we're mapping each doc to right? And so you end up with multiple promises because of how map works? So for each doc, run the _gateway.find(doc.carId) promise (function)

PositiveGuy Over a year ago

I'll probably end up using a library but I bet you can tell me more on why I'd want to use a library (what does it give me vs. out of box vanilla promises?)

greim Over a year ago

Yep, that's exactly how map() works :) As far as something beyond vanilla promises, I'd strongly recommend "co" or something similar: npmjs.com/package/co

Collectives™ on Stack Overflow

Best way to iterate and make an async call during each iteration

2 Answers 2

1 Comment

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related