Stream JSON with RxJS Observable

Question

I'm trying to understand a few things about RxJs. What I would like to do is consume some JSON data and immediately begin to render that data on the DOM as it's coming in. I've setup the stream request, response, and display. It's outputting every just fine but it's doing it all at once and not over time.

I want to start showing the data on the page as its coming in, instead of waiting for the whole file to complete then show at once which would create a long wait time.

//Cache the selector
var $resultList = $('.results');

//Gets the JSON (this will not be a static file,just for testing)
var requestStream = Rx.Observable.just("/results.json");

var responseStream = requestStream
    .flatMap(function(requestUrl) {
            return Rx.Observable.fromPromise(jQuery.getJSON(requestUrl))
             });

var displayStream = responseStream.subscribe(
    function(response) {
    //This maps to a Handlebars Template and puts it on the DOM
    $resultList.html(compiledTemplate(response)); 
            },
            function(err) {
                    console.log('Error: %s', err);
             },
             function() {
                    console.log('Completed');
             });




//Sample of the data from the JSON file
Object{
    beginIndex: "1"
    catId: "111"
    endIndex: "1"
    products: Array[100]

}

You have everything set up correctly, however, getJSON retrieves everything at once. In that sense, responseStream isn't a stream, its a fully formed object. And the promise resolved from jQuery.getJSON is just the entire response from the page. Your code would produce the actions you're looking for only if you were loading data from multiple sources at once (results.json, results2.json etc). — bstockwell
– bstockwell, Commented Dec 19, 2015 at 18:04
Is there a way to begin converting the single JSON file into an object and start looping through the Array on that object as its coming in? I'm looking for a way to stream the data incrementally as it's coming in and being processed. — agdev
– agdev, Commented Dec 19, 2015 at 18:33
You're really dependent on the behavior of the browser. The response comes in chunks which have a uniform size, but could begin and end anywhere in the data objects: [chunk 1, data: "[{object 1},{[ob] [chunk 2, data: "[{ject2}{[object3...etc.... The broswer collect these and puts them in the right order. Now what you're asking about is getting into hacking the http protocol, which is overkill. If you want to parse this object by object, you'll want to connect to your server with a websocket and emit each member one by one, which is more streamy than a single get call. — bstockwell
– bstockwell, Commented Dec 19, 2015 at 18:43
Strike that last answer, it's probably sending everything in 1 big chunk, which, for our purposes is going to arrive at an instant in time. You're still looking for a websocket like behavior. (eg: socket.io) — bstockwell
– bstockwell, Commented Dec 19, 2015 at 18:47
So if streaming is not a good route, how about chunking? So take the first 10 in that array and send that off to be displayed. Then the next 10 and so on. So more like buffering/processing that in smaller chunks. — agdev
– agdev, Commented Dec 19, 2015 at 18:51

user3743222 · Accepted Answer · 2015-12-19 19:37:20Z

5

If I understand well, there are two relevant points to make:

you need to find a way to have a stream of objects from that file instead of one big object when you finish reading that file (I want to start showing the data on the page as its coming in). The mechanics of that would depend first on the structure of the source (file and file reading mechanism) than on Rxjs (is every line an object that can lead to information display etc.?). Once you have that 'minimum displayable unit of information' you can use Rxjs to buffer/process it if need be (do you want to display something for each object, or each 100 objects, or remove uncessary attributes etc.?)
you need to update your display incrementally as new data arrive. That means you need something like $resultList.html($resultList.html() + compiledTemplate(response)); to append the new compiled html to the old one.

UPDATE : for chunking an array, you can have a look at this jsfiddle : http://jsfiddle.net/429vw0za/

var ta_result = document.getElementById('ta_result');

function emits ( who, who_ ) {return function ( x ) {
 who.innerHTML = [who.innerHTML, who_ + " emits " + JSON.stringify(x)].join("\n");
};}

function fillArrayWithNumbers(n) {
        var arr = Array.apply(null, Array(n));
        return arr.map(function (x, i) { return {prop1: i, prop2:i, prop3:i} });
    }

var sampleObj = {
    beginIndex: "1",
    catId: "111",
    endIndex: "1",
    products: fillArrayWithNumbers(100)
}

console.log('sampleObj', sampleObj);

var result$ = Rx.Observable
  .from(sampleObj.products)
  .bufferWithCount(10)
  .map(function(mini_array){return {
  beginIndex: sampleObj.beginIndex,
  catId: sampleObj.catId,
  endIndex: sampleObj.endIndex,
  products: mini_array
  }})
  .do(emits(ta_result, 'result'));

result$.subscribe(function(){    });

You will then have a stream of objects with arrays of size 10 taken from the array of size 100.

edited Dec 19, 2015 at 19:37

answered Dec 19, 2015 at 18:05

user3743222

18.7k5 gold badges74 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

agdev Over a year ago

Hi thank you for your response. 1.) I need to loop through the products Array on that object and pull in the other 3 properties on that object for the Handlebars template. So there will be one large object to consume.

user3743222 Over a year ago

Is your array so huge that it makes sense to chunk it (or the processing time of one item so long)? With Rxjs, you can easily chunk an array into a stream of 'mini'-arrays. So you could get that large object and output a stream of mini-arrays that you consume through an array-adjusted version of your current code. Would that work for you?

agdev Over a year ago

Yes, that would be perfect. If I could stream the chunks of that array to be displayed then I could get products on the page quicker rather than waiting for all 100 to be displayed at the same time.

user3743222 Over a year ago

answer using Rxjs posted. However, I wonder if you really have to use Rxjs. You could directly slice your array and do a simple loop. So you would not need streams at all here, just regular promises and regular synchronous code. Also : flatMap accepts promises, so you can write directly return jQuery.getJSON(requestUrl)

tomriddle_1234 Over a year ago

In RxJS 5, bufferWithCount changed to bufferCount. github.com/ReactiveX/rxjs/blob/master/MIGRATION.md. Trouble...

|

Collectives™ on Stack Overflow

Stream JSON with RxJS Observable

1 Answer 1

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related