44

I'm trying to make simple feed reader in node and I'm facing a problem with multiple requests in node.js. For example, I got table with urls something like:

urls = [
"http://url1.com/rss.xml",
"http://url2.com",
"http://url3.com"];

Now I want to get contents of each url. First idea was to use for(var i in urls) but it's not good idea. the best option would be to do it asynchronously but I don't know how to make it.

Any ideas?

EDIT:

I got this code:

var data = [];
for(var i = 0; i<urls.length; i++){
    http.get(urls[i], function(response){
    console.log('Reponse: ', response.statusCode, ' from url: ', urls[i]);
    var body = '';
    response.on('data', function(chunk){
        body += chunk;
    });

    response.on('end', function() {
        data.push(body);
    });
}).on('error', function(e){
    console.log('Error: ', e.message);
});
}

Problem is that first is call line "http.get..." for each element in loop and after that event response.on('data') is called and after that response.on('end'). It makes mess and I don't know how to handle this.

Improve this question
4
  • What is the question? What are you trying to achieve? Commented Nov 11, 2013 at 16:41
  • Do not use for (x in y) on arrays! It does not do what you think! Use for (idx=0,len=array.length; idx < len; ++idx){value=array[idx]; ....}. Commented Nov 11, 2013 at 16:43
  • 1
    To get contents of each urls asynchronously in one loop Commented Nov 11, 2013 at 16:44
  • This is a duplicate question, answered many times on stack overflow. Commented Nov 11, 2013 at 18:51

6 Answers 6

Reset to default
55

I know this is an old question, but I think a better solution would be to use JavaScripts Promise.all():

const request = require('request-promise');
const urls = ["http://www.google.com", "http://www.example.com"];
const promises = urls.map(url => request(url));
Promise.all(promises).then((data) => {
    // data = [promise1,promise2]
});
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

why would it be better? Little explanation would really make it better!
I like this approach. Much cleaner
@MdSifatulIslam this uses es6 so it may not be helpful for everyone. It's more concise, efficient, and simpler to digest once you're used to this style of [modern] coding.
do you know what happens if the response type of those requests are of type NodeJS.ReadableStream|FileObject|Buffer as is the case with watson text to speech api response? are the responses fully "received" and "completed" at the time of the Promise.all() callback? when i try to pipe each readable stream response into a writeable stream within a forEach() loop over the promise responses in the callback, i get incomplete files.
As of Feb 11th 2020, request is fully deprecated. - github.com/request/request-promise
47

By default node http requests are asynchronous. You can start them sequentially in your code and call a function that'll start when all requests are done. You can either do it by hand (count the finished vs started request) or use async.js

This is the no-dependency way (error checking omitted):

var http = require('http');    
var urls = ["http://www.google.com", "http://www.example.com"];
var responses = [];
var completed_requests = 0;

for (i in urls) {
    http.get(urls[i], function(res) {
        responses.push(res);
        completed_requests++;
        if (completed_requests == urls.length) {
            // All download done, process responses array
            console.log(responses);
        }
    });
}
Improve this answer

5 Comments

thanks, it inspired me to solve this problem just the way you said.
actually, the way it is right now, it will say it is complete when the last request is send, but NOT when all the request are done. I gave an answer of what I ended up doing to solve this.
Adrian's comment is very important. Be aware of that this answer does not mean that any of those requests completed. By this solution, you only know requests were 'sent' and iteration was over.
Shouldn't it be "http.get(urls[url]..." or in that case "for (url of urls)"? assuming that urls is an array.
what if i wanted to perform multiple http requests with arrays and they have to be executed sequentially..I tried async /await to but they are executed parallely...What i mean by sequentially is,after the completion of one array of http requests,it should goto the next array of http requests.Async/await executes all arrays of HTTP requests simultaneously
28

You need to check that on end (data complete event) has been called the exact number of requests... Here's a working example:

var http = require('http');
var urls = ['http://adrianmejia.com/atom.xml', 'http://twitrss.me/twitter_user_to_rss/?user=amejiarosario'];
var completed_requests = 0;

urls.forEach(function(url) {
  var responses = [];
  http.get(url, function(res) {
    res.on('data', function(chunk){
      responses.push(chunk);
    });

    res.on('end', function(){
      if (completed_requests++ == urls.length - 1) {
        // All downloads are completed
        console.log('body:', responses.join());
      }      
    });
  });
})
Improve this answer

3 Comments

How it would work using request module instead of http? I mean is it possible to achive this using request and async modules?
@Adrian shouldn't var responses = []; be declared outside the forEach instead of inside ?
@blue-sky no, because I want to keep the response from each link separate. The forEach loop will start with the first link and data will push multiple chunks. end will display the full response. On the next link, I want to start with an empty response, since I already logged out the response.
1

You can use any promise library with ".all" implementation. I use RSVP library, Its simple enough.

var downloadFileList = [url:'http://stuff',dataname:'filename to download']
var ddownload = downloadFileList.map(function(id){
          var dataname = id.dataname;
          var url = id.url;
          return new RSVP.Promise(function(fulfill, reject) {
           var stream = fs.createWriteStream(dataname);
            stream.on('close', function() {
            console.log(dataname+' downloaded');
            fulfill();  
            });
          request(url).on('error', function(err) {
    console.log(err);
    reject();
  }).pipe(stream);
        });
        });      
        return new RSVP.hashSettled(ddownload);
Improve this answer

Comments

1

Promise.allSettled will not stop at error. It make sure you process all responses, even if some have an error.

Promise.allSettled(promises)
 .then((data) => {
// do your stuff here
 })
 .catch((err) => {
   console.log(JSON.stringify(err, null, 4));
 });
Improve this answer

Comments

0

The problem can be easily solved using closure. Make a function to handle the request and call that function in the loop. Every time the function would be called, it would have it's own lexical scope and using closure, it would be able to retain the address of the URL even if the loop ends. And even is the response is in streams, closure would handle that stuff too.

const request = require("request");

function getTheUrl(data) {
    var options = {
        url: "https://jsonplaceholder.typicode.com/posts/" + data
    }
    return options
}

function consoleTheResult(url) {
    request(url, function (err, res, body) {
        console.log(url);
    });
}

for (var i = 0; i < 10; i++) {
    consoleTheResult(getTheUrl(i))
}
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.