Issue trying to run synchronous REST requests in Node

Question

First of all, I'm not experienced in asynchronous programming, so I'm sorry if I missed something obvious.

I see this question pop up a lot. I know that people don't like forcing synchronicity in Javascript, but it is necessary in this case. I'm making heavy calls to a production database that can't take too much load due to how frequently it is used. Because of this, I'm setting up my code in a loop to make a request, wait for confirmation that it is finished, sleep for 2 seconds, and then make the next request. This is because I'm going to be pulling a LOT of data from this server on a weekly basis over the course of around 10-20 minutes.

Here's the code that I have. Sleep is a function that forces the program to wait using the Date class.

    var thread = function(cb){
        cb();

    };


    do{

        var x = thread(function(){
            request.post(options, function(e, r, body){
                console.log(e);
                console.log(r.statusCode);
                issues.push(body["issues"]);
                maxResults = body["total"];
                options.body.startAt += 25;
                console.log("Going to sleep");

            });
            sleep(2000);
        });
        console.log("Waking up and moving to the next cycle");

    }while(issues.length < maxResults);
    console.log("Finished with the requests");
}

although I have a callback set up, my code is still running the requests asynchronously. Because I leave maxResults null, it is plainly obvious that my callback isn't working. This is my output:

Waking up and moving to the next cycle
Finished with the requests
Going to sleep

I constantly see people say that, but I haven't been able to find the correct way to structure a promise for this purpose, even after reading through the bluebird readme. It looked to me like promises were more of a way to chain values through functions than to act as a join for async threads like I want to use them for. Could you please give me an example? — John Kulp
– John Kulp, Commented Jul 2, 2015 at 17:18
Thanks for the help, but that example doesn't really apply to my situation. I'm basing the while condition of the loop on the result of each post call. Because of this, I can't exactly put all of my async functions into a list to resolve in order at some time in the future. I need to resolve each function after I create it. I've been looking for a while and I still can't find out how to force the resolution of a function with bluebird. This may not conform to async coding conventions, but its essential for the database that I'm working with. Do I need to make a 1000 deep recursive call? — John Kulp
– John Kulp, Commented Jul 2, 2015 at 17:54

SLaks · Accepted Answer · 2015-07-02 17:58:06Z

1

You need to make a recursive asynchronous function.

It would look something like this:

function fetch(existingIssues) {
    return sendRequest().then(function() { 
        existingIssues.push(...);
        if (existingIssues.length >= maxResults)
            return existingIssues;
        else
            return fetch(existingIssues);
    });
}
fetch([]).then(...);

answered Jul 2, 2015 at 17:58

SLaks

891k182 gold badges1.9k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

John Kulp Over a year ago

This seems very dangerous. I'm working with large objects, table rows with ~30 columns including description and summary columns. This needs to be able to handle ~5-10 GB of data in total, so a recursive function would quickly become bloated and slow. I mean, if this is the only way, its the only way... but is there no other solution?

SLaks Over a year ago

@user3723527: No. Asynchronous recursion does not bloat the call stack; each async boundary starts with an empty call stack. There is nothing wrong with this.

John Kulp Over a year ago

I'll take your word for it, but I'm curious about how that works. If I have the async function var foo = function(){ var x = 2 ; return x + foo();} wouldn't that populate the stack with infinite variables set to 2? How does Javascript avoid this? Is it smart enough to differentiate and garbage collect the functions that drop off after the recursive call or don't use scoped variables in the return value?

SLaks Over a year ago

@user3723527: That's not async. The whole point of async is that the callback runs later, which is, by definition, after the stack frame has exited. And there is no closured relationship across async calls; that only happens with lexical nesting. You should learn about the Node.js event loop.

Collectives™ on Stack Overflow

Issue trying to run synchronous REST requests in Node

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related