Request an external streaming JSON feed in Node

Question

In Node.js I'm using stream-chain and stream-json to request streaming feeds from local and remote resources. Below works for local resources, but how to modify it, to allow also for external resources? Do I need to download the file first or?

const fs              = require('fs');
const { chain }       = require('stream-chain');
const Pick            = require('stream-json/filters/Pick');
const { streamArray } = require('stream-json/streamers/StreamArray');

const path = './feed.json'; // External URL or local file

const pipeline = chain([
    fs.createReadStream(path),
    Pick.withParser({ filter: 'products', once: true }), // Custom: modify feed
    streamArray(),
    data => data
]);

pipeline.on('data', () => {
    counter++;
    console.log(data);
});
pipeline.on('error', error => console.log(error));
pipeline.on('end', () => console.log(`Found ${counter} entries`));

You mean an http url with a stream feed?

Tarun Lalwani
– Tarun Lalwani

2019-06-25 03:11:59 +00:00
Commented Jun 25, 2019 at 3:11 — Tarun Lalwani
– Tarun Lalwani, Commented Jun 25, 2019 at 3:11
what are you trying to achieve?

dawsnap
– dawsnap

2019-06-26 14:03:27 +00:00
Commented Jun 26, 2019 at 14:03 — dawsnap
– dawsnap, Commented Jun 26, 2019 at 14:03

3960278 · Accepted Answer · 2019-06-30 01:05:05Z

3

+50

The question boils down to having a readable stream through network and processing it on run time.

I don't think this is going to work, and, finally you have to download the file only, and process as local file.

There are ways to get the file from network:

var remotePath = "https://....."
https.get(remotePath, response => {
  var stream = response.pipe(file);

  stream.on("finish", function() {
    console.log("done");
  });
});

or using request"

request(remotePath).pipe(fs.createWriteStream('./remoteFeed.json'))

But at end there is no readable stream with run time processing.

answered Jun 30, 2019 at 1:05

3960278

8167 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Emmanuel Ani · Accepted Answer · 2019-07-15 13:10:31Z

1

I think you need to download the file and process manually. You can use request() function to get online file. But i agree with Suryapratap. but you can also write to a file and read from same file and continue with your program. example

    request("https://reqres.in/api/users",(err, res, body)=>{
        if(err) console.log(err);
    const objectBody = JSON.parse(body)
    fs.writeFile('justtest.json',JSON.stringify(objectBody),err=>{
     if(err) throw err;
    })

     const readable = fs.createReadStream('./justtest.json', "UTF-8");

     readable.on("data", data =>{
           payload = JSON.parse(data);
           console.log(payload);
      } )

 })

you can now use object to your stuff.

edited Jul 15, 2019 at 13:10

answered Jul 1, 2019 at 8:36

Emmanuel Ani

4726 silver badges14 bronze badges

Collectives™ on Stack Overflow

Request an external streaming JSON feed in Node

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related