nodejs request - streaming a response file

Question

In my nodejs app, I am trying to read a http file url, and then download the contents of that file in a streaming manner.

What I want to do is: - make to request to the file resource (using node request module) - when response starts to become available, then start reading the data in chunks, rather than have the file downloaded to disk..

I understand that the request module does support streaming, and I have verified that the below code works:

var request = require('request');

var fileUrl = "http://172.19.2.48:8080/files/1472628282006/valid.txt";

request(fileUrl, function(err, response, body) {})
.on('response', function(response) {

  /*
  response.on('readable', function() {
    console.log("now readable");
    var ch;
    while ((ch=response.read(1))!== null) {
      console.log("char:", ch);
    }
  });
  */

  response.on('data', function(data) {
    console.log('data: ', data.toString());
  });

});

But only problem is, I don't have control on "reading how much I want to read", since the 'data' event gives whatever available at that point of time. Instead what I wanted to do was something like using a read operation myself, as in the commented code in the above snippet.

Such code generally works for nodeJS steams 2, but I cannot get it working here. The readable event is fired, but the read operation returns null. My use case is, I am going to read some sort of structured data, and I am gonna parse it by reading a char at a time using some finite state machine.

So, is there any to read, rather than be notified by a 'data' event?

output, when trying to read. Only readable event is received, subsequent reads return null.

rvnath@admin ~/downloader $ node download.js
now readable

Edit In a nutshell, I want to be able to use the incoming response steam in a streams2 (pull based stream) manner, rather than streams 1(push based) type.

Vasan · Accepted Answer · 2016-08-31 11:17:51Z

2

The problem is, the stream is in flowing mode so all data is read automatically. Maybe you can try this:

request(fileUrl, function(err, response, body) {})
.on('response', function(response) {
  response.pause(); // <-- Pauses stream so you can pull data on demand
  response.on('readable', function() {
    console.log("now readable");
    var ch;
    while ((ch=response.read(1))!== null) {
      console.log("char:", ch);
    }
  });
});

edited Aug 31, 2016 at 11:17

answered Aug 31, 2016 at 11:09

Vasan

4,9564 gold badges22 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

nodejs request - streaming a response file

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related