Node.js AWS Lambda + Archiver lib - Error in zip file creation

Question

everyone,

I have written a lambda in Node.js that takes a folder and zips its contents. To do this I used the Archiver library: Archiver

The following is my code to create the File:

const zipFilePath = '/tmp/' + 'filename' + '.zip'
const output = fs.createWriteStream(zipFilePath);

const archive = archiver('zip', {
    zlib: { level: 9 }
});

output.on('close', function () {
    console.log(archive.pointer() + ' total bytes');
    console.log('archiver has been finalized and the output file descriptor has closed.');
});

archive.on('error', function(err){
    throw err;
});

await archive.pipe(output);
await archive.directory(destFolder, false);
await archive.finalize();

To write the files I am using the /tmp folder of the lambdas, which is the only folder with write permissions.

The flow is as follows:

I get the path to the folder
Zip the content and save it in the folder destFolder

The file is then saved on an S3 bucket:

const file = await fs.readFileSync(filePath)
const params = {
    Bucket: bucketName,
    Key: fileName,
    Body: file
};
const res = await s3.upload(params).promise()
return res.Location

The zip file is generated, the problem is that when I download it, it is corrupted. I tried analysing it with an online zip file analyser (this) and the result of the analysis is the following:

Type = zip
ERRORS:
Unexpected end of archive
WARNINGS:
There are data after the end of archive
Physical Size = 118916
Tail Size = 19
Characteristics = Local

and the analyser shows that the files are all present in the .zip (I can see their names).

The strangest thing is that if instead of .zip the file (again with the same library) I create it in .tar

const archive = archiver('tar', {
    zlib: { level: 9 }
});

The file is generated correctly and I can extract it as an archive. Basically, it is as if something is wrong with the .zip format alone.

Has anyone ever experienced a similar episode? Can you help me find a solution? I need to create files in .zip format. Thank you.

valdeci · Accepted Answer · 2022-02-17 15:52:09Z

1

The problem is that you are not being able to zip your files properly, this can occurs by many issues, including:

You are not waiting the file to be processed, you need to use .close() event to do this.
You are not sending the correct path to the files or dirs to zipped, normally the files that you upload together with the lambda files on the root dir of your project stay on /var/task/ on the Lambda dir system, so to send the correct file use __dirname + '/file.name'
You are not appending correctly the files, check the .file() and .append() methods if you are sending the files correctly

If you have the following Lambda structure:

~/my-function
├── index.js
└── node_modules
├── package.json
├── package-lock.json
├── test.txt //file to be sent zipped on s3

The following example will works for you:

const archiver = require('archiver')
const fs = require('fs')
const AWS = require('aws-sdk');
const s3 = new AWS.S3({apiVersion: '2006-03-01'});


const sendToS3 = async (filePath) => {
    const bucketName = "bucket_name";
    const fileName = "zipped.zip"
    const file = await fs.readFileSync(filePath)
    const params = {
        Bucket: bucketName,
        Key: fileName,
        Body: file
    };
    const res = await s3.upload(params).promise()
    return res.Location
}

exports.handler = async event => {
  return new Promise((resolve, reject) => {
    const zippedPathName = '/tmp/example.zip';
    const output = fs.createWriteStream(zippedPathName);
    const fileToBeZipped = __dirname + '/test.txt';
    const zip = archiver('zip')

    zip
      .append(fs.createReadStream(fileToBeZipped), { name: 'test.txt' })
      .pipe(output)

    zip.finalize()

    output.on('close', (result => {
        sendToS3(zippedPathName).then(result => {
            resolve(result)
        })
    }))
  })
}

edited Feb 17, 2022 at 15:52

answered Feb 17, 2022 at 14:07

valdeci

15.6k7 gold badges64 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

AndreC23 Over a year ago

Hi @valdeci, your solution works perfectly! I modified the code a bit because I needed to zip a directory containing files and not a single file, so I used the .directory() method instead of .append(). However, I kept the /tmp folder of the Lambdas because from what I have read in the documentation it is the only writable folder. Do you know if /var/task is also writable? Do you know if the /tmp folder is automatically emptied after execution? Thanks again!

valdeci Over a year ago

The /var/task is the directory where function files are saved, the directory is not writable (eg. if you upload a test.txt with the code that you upload for your function will be able to read it), the /tmp is only writable directory that you have in the Lambda function

AndreC23 Over a year ago

thank you! That was really what i was looking for.

Collectives™ on Stack Overflow

Node.js AWS Lambda + Archiver lib - Error in zip file creation

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related