node - send large JSON over net socket

Question

The problem is that sending large serialized JSON (over 16,000 characters) over a net socket gets split into chunks. Each chunk fires the data event on the receiving end. So simply running JSON.parse() on the incoming data may fail with SyntaxError: Unexpected end of input.

The work around I've managed to come up with so far is to append a null character ('\u0000') to the end of the serialized JSON, and check for that on the receiving end. Here is an example:

var partialData = '';
client.on( 'data', function( data ) {
    data = data.toString();
    if ( data.charCodeAt( data.length - 1 ) !== 0 ) {
        partialData += data;
        // if data is incomplete then no need to proceed
        return;
    } else {
        // append all but the null character to the existing partial data
        partialData += data.substr( 0, data.length - 1 );
    }
    // pass parsed data to some function for processing
    workWithData( JSON.parse( partialData ));
    // reset partialData for next data transfer
    partialData = '';
});

One of the failures of this model is if the receiver is connected to multiple sockets, and each socket is sending large JSON files.

The reason I'm doing this is because I need to pass data between two processes running on the same box, and I prefer not to use a port. Hence using a net socket. So there would be two questions: First, is there a better way to quickly pass large JSON data between two Node.js processes? Second, if this is the best way then how can I better handle the case where the serialized JSON is being split into chunks when sent?

Curious..why would you want to do this over socket connection vs HTTP? — ming_codes
– ming_codes, Commented May 7, 2012 at 21:10
having one socket file broadcaster I can create many "listeners" that grab the data then do stuff with it (e.g. send it to a database, graph on a browser, etc.). though I am open to any method of more reliably sending data from one to many Node.js processes. — Trevor Norris
– Trevor Norris, Commented May 7, 2012 at 21:12

alex · Accepted Answer · 2012-05-09 05:56:12Z

4

You can use try...catch every time to see if it is a valid json. Not very good performance though.
You can calculate size of your json on sending side and send it before JSON.
You can append a boundary string that's unlikely be in JSON. Your \u0000 - yes, it seems to be a legit way. But most popular choice is newline.
You can use external libraries like dnode which should already do something I mentioned before. I'd recommend trying that. Really.

One of the failures of this model is if the receiver is connected to multiple sockets, and each socket is sending large JSON files.

Use different buffers for every socket. No problem here.

answered May 9, 2012 at 5:56

alex

12.3k3 gold badges33 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

UndeadKernel · Accepted Answer · 2013-11-11 22:59:26Z

1

It is possible to identify each socket individually and build buffers for each one. I add an id to each socket when I receive a connection and then when I receive data I add that data to a buffer.

net.createServer( function(socket) {
  // There are many ways to assign an id, this is just an example.
  socket.id = Math.random() * 1000;
  socket.on('data', function(data) {
    // 'this' refers to the socket calling this callback.
    buffers[this.id] += data;
  });
});

Each time you can check if you have received that "key" delimiter that will tell you that a buffer is ready to be used.

answered Nov 11, 2013 at 22:59

UndeadKernel

5335 silver badges16 bronze badges

Collectives™ on Stack Overflow

node - send large JSON over net socket

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related