Confused about Stream implementation in NodeJs

I'm not sure on the exact details of the protocol that you're trying to implement but the following should give you a good pattern that you can adapt to your needs.

My fictional protocol

When a client socket connects to my TCP server, I want to send them a file. But first I want to send a header. At the end of the file, before the stream ends, I also want to send a header. So the data written to the socket looks like:

==== HEADER ====
[FILE CONTENTS]
==== FOOTER ====

Implementing a Transform stream

All I want to do is transform the data coming out of a readable stream. Notice transform is the keyword here. I can use a Transform stream for this.

When creating a Transform stream, you can override two methods: _transform and _flush. _transform is called with each chunk coming of the readable stream. You can change the data, buffer it up or whatever. _flush is called right after all the data from the readable has finished. You can do anymore cleanup here, or write a last bit of data out.

var Stream = require('stream');
var Util = require('util');

var MyProtocolStream = function () {

    this.writtenHeader = false;            // have we written the header yet?
    Stream.Transform.call(this);
};

Util.inherits(MyProtocolStream, Stream.Transform);

MyProtocolStream.prototype._transform = function (chunk, encoding, callback) {

    if (!this.writtenHeader) {
        this.push('==== HEADER ====\n');  // if we've not, send the header first
        this.writtenHeader = true;
    }
    this.push(chunk);                     // send the incoming file chunks along as-is
    callback();
};

MyProtocolStream.prototype._flush = function (callback) {

    this.push('==== FOOTER ====\n');      // Just before the stream closes, send footer 
    callback();
};

Using the MyProtocolStream

So now I have a stream that does what I want, I can simply pipe a file (or any readable stream for that matter) through my custom Transform stream and into any other Writable stream (such as a socket).

var Fs = require('fs');
var Net = require('net');

var server = Net.createServer(function (socket) {

    Fs.createReadStream('./example.txt')
        .pipe(new MyProtocolStream())
        .pipe(socket);
});

server.listen(8000);

Testing it out

I can test this out by adding some contents to example.txt:

This is a line
This is another line
This is the last line

I can spin up my server and then connect with telnet/nc:

$ telnet 127.0.0.1 8000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
==== HEADER ====
This is a line
This is another line
This is the last line
==== FOOTER ====
Connection closed by foreign host.

So what about Duplex streams?

A duplex stream is two streams embedded within one. Data comes out of one and you write totally different data into the other one. It's used where there's 2-way communication with another entity (such as a TCP socket). In this example, we don't need a Duplex stream because data is only flowing in one direction:

file -> MyProtocolStream -> socket

Learning more

As Meir pointed out in the other answer, Substack's stream handbook is the canonical (and best) resource for streams along with the official docs. If you read them thoroughly and implement the examples yourself, you'll learn all you need to know about streams.

Sending multiple files over a single socket in series

If you're wanting to write the output of multiple of these Transforms streams into a single writable end, pipe() isn't going to work for you. Once the EOF comes from a single stream, the upstream writable (socket) is also going to be closed. Also there's no guarantee about ordering of data events in this case. So you need to manually aggregate the streams by listening to data/end events, starting to read one stream after another has finished:

var server = Net.createServer(function (socket) {

    var incoming1 = Fs.createReadStream('./example.txt')
            .pipe(new MyProtocolStream());

    var incoming2 = Fs.createReadStream('./example.txt')
            .pipe(new MyProtocolStream());

    var readStream = function (stream, callback) {

        stream.on('data', socket.write.bind(socket));
        stream.on('end', callback);
    };

    readStream(incoming1, function () {

        readStream(incoming2, function () {

            socket.end();
        });
    });
});

If you're nested callback-averse, you could also use promises:

var server = Net.createServer(function (socket) {

    var incoming1 = Fs.createReadStream('./example.txt')
            .pipe(new MyProtocolStream());

    var incoming2 = Fs.createReadStream('./example.txt')
            .pipe(new MyProtocolStream());

    var incoming3 = Fs.createReadStream('./example.txt')
            .pipe(new MyProtocolStream());

    var readStream = function (stream) {
        return new Promise(function (resolve, reject) {
            stream.on('data', socket.write.bind(socket));
            stream.on('end', resolve);
        });
    };

    readStream(incoming1)
    .then(function () {
        return readStream(incoming2);
    })
    .then(function () {
        return readStream(incoming3);
    })
    .then(function () {
        socket.end();
    });
});