2

So I am writing my web server in pure node.js, only use bluebird for promisify. This has been bothering me for a week, and I can't decide which one should I really use. I have read tons of posts, blogs and docs on these two topic, please answer based on your own working experience, thanks. Here goes the detailed sum up and related questions.

Both approach has been tested, they both work great. But I can't test for performance, I only have my own basic website files(html, css, img, small database, etc.), and I never have managed video files and huge databases.

Below are code parts, to give you basic ideas(if you really know which one to use, don't bother reading the code, to save you some time), this question is not about logic, so you can just read the parts between dash lines.

About fs.createReadStream:
Pros: good for huge files, it reads a chunk at a time, saves memory and pipe is really smart.
Cons: Synchronous, can't be promisified(stream is a different concept to promise, too hard to do, not worth it).

//please ignore IP, its just a custom name for prototyping.
IP.read = function (fpath) {
    //----------------------------------------------------
    let file = fs.createReadStream(fpath);
    file.on('error', function () {
        return console.log('error on reading: ' + fpath);
    });
    return file;
    //----------------------------------------------------
};
//to set the response of onRequest(request, response) in http.createServer(onRequest).
IP.setResponse = function (fpath) {
    let ext = path.extname(fpath),
        data = IP.read(fpath);
    return function (resp) {
        //----------------------------------------------------
            //please ignore IP.setHeaders.
        resp.writeHead(200, IP.setHeaders(ext));
        data.pipe(resp).on('error', function (e) {
            cosnole.log('error on piping ' + fpath);
        });
        //----------------------------------------------------
    }
};

About fs.readFile:
Pros: Asynchronous, can easily be promisified, which makes code really easy to write(develop) and read(maintain). And other benefits I haven't gotten a hand on yet, like data validation, security, etc.
Cons: bad for huge files.

IP.read = function (fpath) {
    //----------------------------------------------------
    let file = fs.readFileAsync(fpath);
    return file;
    //----------------------------------------------------
};
//to set the response of onRequest(request, response) in http.createServer(onRequest).
IP.setResponse = function (fpath) {
    const ext = path.extname(fpath);
    return function (resp) {
        //----------------------------------------------------
        IP.read(fpath).then((data) => {
            resp.writeHead(200, IP.setHeaders(ext));
            resp.end(data);
        }).catch((e) => {
            console.log('Problem when reading: ' + fpath);
            console.log(e);
        });
        //----------------------------------------------------
    }
};

Here are my options:
• The easy way: Using fs.createReadStream for everything.
• The proper way: Using fs.createReadStream for huge files only.
• The practical way: Using fs.readFile for everything, until related problems occur, then handle those problems using fs.createReadStream.

My final decision is using fs.createReadStream for huge files only(I will create a function just for huge files), and fs.readFile for everything else. Is this a good/proper decision? Any better suggestions?

P.S.(not important):
I really like building infrastructure on my own, to give you an idea, when I instantiate a server, I can just set the routes like this, and customize whatever or however I want. Please don't suggest me to use frameworks:

let routes = [
    {
        method: ['GET', 'POST'],
        uri: ['/', '/home', '/index.html'],
        handleReq: function () {return app.setResp(homeP);}
    },
    {
        method: 'GET',
        uri: '/main.css',
        handleReq: function () {return app.setResp(maincssP);}
    },
    {
        method: 'GET',
        uri: '/test-icon.svg',
        handleReq: function () {return app.setResp(svgP);}
    },
    {
        method: 'GET',
        uri: '/favicon.ico',
        handleReq: function () {return app.setResp(iconP);}
    }
];

Or I can customize it and put it in a config.json file like this:

{
"routes":[
    {
        "method": ["GET", "POST"],
        "uri": ["/", "/home"],
        //I will create a function(handleReq) in my application to handle fpath
        "fpath": "./views/index.html"
    },
    {
        "method": "GET",
        "uri": "/main.css",
        "fpath": "./views/main.css"
    },
    {
        "method": "GET",
        "uri": "/test-icon.svg",
        "fpath": "./views/test-icon.svg"
    }
]
}
4
  • 1
    What would be the problem exactly of always using fs.createReadStream()? It works, it's easy to use, it's much better for memory consumption, and the streaming itself isn't synchronous so it won't block your app. Commented Jun 19, 2016 at 7:07
  • Related: github.com/petkaantonov/bluebird/issues/225 - rl;dr: You don't need to promisify createReadStream() as it returns immediately anyway. Don't overuse promises "just because". Commented Jun 19, 2016 at 7:30
  • To robertklep: as I have stated, there are no problems at all for both implementations. Thanks quote from you, "the streaming itself isn't synchronous so it won't block your app", really has cleared things up. And yeah, I am going for the easy way. Commented Jun 19, 2016 at 7:45
  • To Tomalak: yeah, I have read that post couple days ago. And I understand that promise returns one result only, unlike stream's behavior, results over time. Thanks for reminding me not to over use promises just because I feel it's cool. Commented Jun 19, 2016 at 7:49

1 Answer 1

2

Let's discuss the actual practical way.

You should not be serving static files from Node.js in production

createReadStream and readFile are both very useful - createReadStream is more efficient in most cases and consider it if you're processing a lot of files (rather than serving them).

You should be serving static files from a static file server anyway - most PaaS web hosts do this for you automatically and if you set up an environment yourself you'll find yourself reverse-proxying node anyway behind something like IIS which should serve static files.

This is only true for static files, again, if you read them and transform them multiple times your question becomes very relevant.

For other purposes, you can use fs.readFileAsync safely

I use readFile a lot for reading files to buffers and working with them, while createReadStream can improve latency - overall you should get similar throughput and the API is easier to work with and more high level.

So in conclusion

  • If you're serving static files and care about performance - don't do it from Node.js in production anyway.
  • If you're transforming files as streams and latency is important, use createReadStream.
  • Otherwise prefer readFile.
Sign up to request clarification or add additional context in comments.

8 Comments

Thanks for the tip about static files. I am not interested in static files though, they are not much fun at all. I mean you can easily build static websites using github, dropbox(all those cloud storage services), etc. I never have built any applications. In node.js, I have found myself building application without my notice, at the beginning, I thought it's just a cooler PHP. Like this web server application, I am actually building something like Apache/Nginx. I am experimenting with static files first, next I am going to do template engine and database interaction, the fun just keeps coming.
For this post, I have decided to go for the easy way, using createReadStream. After all, being consistent is also very important(says from the guy who has trouble being consistent).
Templates should be cached in memory so fs.readFile is entirely fine - consider using a templating engine which does this for you anyway - everything else static on your otherwise dynamic server should still be served from iis/nginx.
Yes, I am considering using handlebars, but first I just want to learn how to build one also. <br />**Can you enlighten me why it's so bad to use node.js for static contents?** If it's too hard to explain, just couple of hints/words will be great. So I can start to implement the related solution in my web server application.
I found something though, like "In terms of performance, conventional web servers like Apache/Nginx use as system call called sendfile(2), which copies the static resources from disk directly to the network card bypassing process memory(RAM)...", I will keep reading and searching. Thanks for the heads up, because nobody has ever told me to look out for this (In real life, I am all by myself).
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.