Use node to create a tarball from memory buffers (non disk) and store result in memory

Question

Say i have the two buffers:

const bufferFile1 = Buffer.from('Hello World!', 'utf-8')
const bufferFile2 = Buffer.from('Hello Again World!', 'utf-8')

How can i create tarball file buffer/stream/blob (not written to disk) where the the two buffers above should be stored as two files in the tarball. I would like to (be able to) pipe the tarball as a response to a request.

I have looked into using the tar package. But this solution requires paths instead of in memory streams/buffers.

Is it at all possible to accomplish this?

PS: I have very little experience with in memory file handling.

The overall project is to create an api endpoint

takes in parameters either as POST body or url query parameters.
creates a set of in memory files with content based on input
creates a tar.gz file based on these files
stores the tar.gz file in mongodb using GridFS

I dont really have a place to store the files temporarily (serverless environment), thats why i want the solution to be completely in memory.

it seems that this package might be useful: npmjs.com/package/tar-stream — johann1301s
– johann1301s, Commented Dec 15, 2021 at 15:15

IcyIcicle · Accepted Answer · 2023-04-09 07:10:03Z

I was able to get it working with archiver and axios.

archiver is a wrapper of tar-stream and zip-stream. So this is possible in a similar manner with tar-stream only, but I didn't get it working.

If archive.read() is giving null you should make sure

archive.read() is inside of archive.on("finish",...
archive.on("finish",... is called before archive.finalize()

Here is my full code snippet:


const axios = require("axios");
const archiver = require("archiver");

const archive = archiver("tar", { gzip: false, store: false });

// data is a string representation of the text.
archive.append(data, { name: "main.txt" });


archive.on("finish", () => {
  const blob = new Blob([archive.read()], { type: "application/x-tar" });
  const formData = new FormData();
  // FormData seems to only accept blob
  formData.append("file", blob);
  axios
    .post(url, formData, {
      // remove this if your file is not in a binary format
      responseType: 'arraybuffer'
    })
    .then((response) => {
      console.log(response.data); // the response and data 
    })
    .catch((error) => {
      console.error(error);
    });
});

// doing this after registering the events or else it will the data might not be present
archive.finalize();

Collectives™ on Stack Overflow

Use node to create a tarball from memory buffers (non disk) and store result in memory

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related