Express node.js - Requesting JSON, processing it asyncronously

Question

With my node.js app, I'm getting my JSON data from a spreadsheet API. It basically returns JSON of the following.

{  
   "status":200,
   "success":true,
   "result":[  
      {  
         "Dribbble":"a",
         "Behance":"",
         "Blog":"http://blog.invisionapp.com/reimagine-web-design-process/",
         "Youtube":"",
         "Vimeo":""
      },
      {  
         "Dribbble":"",
         "Behance":"",
         "Blog":"http://creative.mailchimp.com/paint-drips/?_ga=1.32574201.612462484.1431430487",
         "Youtube":"",
         "Vimeo":""
      }
   ]
}

It's just a dummy data for now but one thing for certain is that, I need to process values (blog URLs) under Blog differently. With the blog url, I need to get Open Graph data so I'm using a module called open-graph-scraper

With data.js I'm getting the whole JSON and it's available in route index.js as data Then I'm processing this data by checking Blog column. If it's a match, I loop the values (blog URLs) through open-graph-scraper module.

This will give me open graph data of each blog url like the following example JSON.

{ 
data:
    { success: 'true',
         ogImage: 'http://davidwalsh.name/wp-content/themes/punky/images/logo.png',
         ogTitle: 'David Walsh - JavaScript, HTML5 Consultant',
         ogUrl: 'http://davidwalsh.name/',
         ogSiteName: 'David Walsh Blog',
         ogDescription: 'David Walsh Blog features tutorials about MooTools, jQuery, Dojo, JavaScript, PHP, CSS, HTML5, MySQL, and more!' },
      success: true 
}

So my goal is to pass this blog JSON as a separate data from the main JSON and put it in the render as a separate object so it's available in view as two separate JSON. But I'm not sure if my approach with getBlogData is correct. I'm not even sure if processing data like this is a good thing to do in a router file. I would appreciate some directions.

index.js

var ogs = require('open-graph-scraper'); 
var data = require('../lib/data.js');

data( function(data) {

    var getBlogData = function (callback) {

        var blogURL = [];

        if (data.length > 0) { 

            var columnsIn = data[0]; 

            for(var key in columnsIn) { 
                if (key === 'Blog') {
                    for(var i = 0; i < data.length; i++) {
                        blogURL += data[i][key];
                    }
                }   
            }
        };

        ogs({ 
            url: blogURL 
        }, function(er, res) { 
            console.log(er, res); 
            callback(res);
        });
    }

    getBlogData( function (blogData) {
       //I want to make this blogData available in render below 
         but don't know how
    });

    router.get('/', function(req, res, next) {
      res.render('index', { 
        title: 'Express', 
        data: data
      });
    });
});

data.js (my module that gets JSON data)

module.exports = function(callback) {

    var request = require("request")
    var url = "http://sheetsu.com/apis/94dc0db4"

    request({
        url: url,
        json: true
    }, function (error, response, body) {

        if (!error && response.statusCode === 200) {

            var results = body["result"];
            callback(results)

        }
    })

}

piemonkey · Accepted Answer · 2015-09-30 09:59:01Z

1

The problem you'll have is that if you do getBlogData asynchronously (and you should, you don't want the client waiting around for all that data to return), by the time you get the data res.render will have already been called. As you can't call res.render again, you have 2 options that come to mind:

You could query for individual blog data from the client. This will result in more back-and-forth between client and server but is a good strategy if you have a lot of entries in your initial data but only want to display a small number.
You could use websockets to send the data to the client as you retrieve it. Look up something like express.io for an easy way to do this.

answered Sep 30, 2015 at 9:59

piemonkey

7513 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Seong Lee Over a year ago

I'm glad you understood my question. Your option 1 seems more doable for my skill level but it will involve a callback function in the view and inside the callback, I will need to parse JSON and bind with HTML, is that right?

piemonkey Over a year ago

You'll need to add a new route to express to res.send the blog data. You'll need to add some Javascript to your page to make the requests and parse the JSON. You can do it using XMLHttpRequest (which supports JSON despite the name) and JSON.parse or if you're using something like Angular or JQuery you can use them and make your life a bit easier.

Seong Lee Over a year ago

So in this route, I use send rather than render and because it's not using render, I need to make a request to fetch this sent JSON. Correct?

Seong Lee Over a year ago

I guess I will still require data.js in this new route and process blog data within the callback of data() which will be pretty much the same as what I have in index.js except that I use res.send

piemonkey Over a year ago

render creates a html page from a template (in your example it finds a template called 'index') and sends it to the client. In your '/' route you should still use render. You'll need a second route that the Javascript within your web page can call, as this will be returning JSON data rather than creating a new html page you want to use send instead of render.

Collectives™ on Stack Overflow

Express node.js - Requesting JSON, processing it asyncronously

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related