NodeJS streaming JSON from python

Question

I'm creating an API with NodeJS. Every request returns a JSON. Some requests call to a python script and I want to send the JSON generated by python to the user. The code would look like:

child = child_process.spawn(cmd, args);

child.stdout.on('data', (chunk) => {
    res.write(chunk);
});

child.on('error', function(err) {
    res.status(500).json({...});
});

child.on('close', (code) => {
    res.end();
});

The problem with this code is that I can't check if the python output is a JSON. Maybe python writes warnings, error...

What can I do to prevent the user will get something different to JSON?.

EDIT

Right now my code is:

var output = [];

command.stdout.on('data', (chunk) => {
    output.push(chunk)
});

command.on('close', (code) => {
    var stdout = output.join('');
    json_cmd = tryParseJSON(stdout)
    if (json_cmd)
        res.send(json_cmd)...
});

But, I don't want to load all the stdout in a variable. But if I don't do that, I can check if the stdout is a JSON. Can I force python to print just json?. Using always json.dumps and a global try catch would be enough?

Thanks.

Just check if it is JSON, your stdout is a string, so parse it with JSON.parse. If it fails it is not a JSON structure — Yerken
– Yerken, Commented Jun 27, 2016 at 9:18
chunk contains a part of the stdout. So, maybe it is not JSON. chunk1 = {...'color':'bl; chunk2 = ue' ...; chunkN= ...}. — J. Kepp
– J. Kepp, Commented Jun 27, 2016 at 11:55

robertklep · Accepted Answer · 2016-06-27 10:28:31Z

1

Instead of using spawn, you probably want to use exec, which will wait for the Python process to exit and provide you with its output (which you can subsequently try to parse as JSON):

const exec = require('child_process').exec;
...
exec(cmdline, (err, stdout, stderr) => {
  if (err) return res.status(500).json({...});
  // Try to parse stdout as JSON:
  try {
    res.json(JSON.parse(stdout));
  } catch(e) {
    res.status(500).json({...});
  }
});

answered Jun 27, 2016 at 10:28

robertklep

205k38 gold badges421 silver badges413 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

J. Kepp Over a year ago

exec has a buffer limit. I don't want size limits in the python output. Also, in that way you will have all the stdout in a variable... I prefer to use chunks.

robertklep Over a year ago

So how did you want to test the validity of the output if you don't first read it?

J. Kepp Over a year ago

Yes, it is not possible. The solution would be to force python to write only json

Rink16 Over a year ago

@robertklep can you please see this issue stackoverflow.com/questions/66236502/…

Rink16 Over a year ago

@robertklep please see my issue

|

Tamas Hegedus · Accepted Answer · 2016-06-27 12:02:52Z

0

So you need a streaming JSON verifier in NodeJS. The problem is that if you start streaming the json data over the network, and find an error in the JSON halfways, you cannot rollback the traffic to the http headers and change 200 ok to 500 ise. If you want to check the output first you have to accumulate it in the server before sending.

Your current code would do the same. If an error event comes the client will receive a 200 ok with a half stdout and a json object at the end. (Or the response object throws an error if you try to set the status code after it has been sent, I don't know how is this case handled.)

answered Jun 27, 2016 at 12:02

Tamas Hegedus

30.1k12 gold badges67 silver badges102 bronze badges

1 Comment

Rink16 Over a year ago

can you please see this issue stackoverflow.com/questions/66236502/…

Collectives™ on Stack Overflow

NodeJS streaming JSON from python

2 Answers 2

6 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related