How to zip files on s3 using lambda and python

Do not write to disk, stream to and from S3

Stream the Zip file from the source bucket and read and write its contents on the fly using Python back to another S3 bucket.

This method does not use up disk space and therefore is not limited by size.

The basic steps are:

• Read the zip file from S3 using the Boto3 S3 resource Object into a BytesIO buffer object
• Open the object using the zipfile module
• Iterate over each file in the zip file using the namelist method
• Write the file back to another bucket in S3 using the resource meta.client.upload_fileobj method

The Code Python 3.6 using Boto3

s3_resource = boto3.resource('s3')
zip_obj = s3_resource.Object(bucket_name="bucket_name_here", key=zip_key)
buffer = BytesIO(zip_obj.get()["Body"].read())

z = zipfile.ZipFile(buffer)
for filename in z.namelist():
    file_info = z.getinfo(filename)
    s3_resource.meta.client.upload_fileobj(
        z.open(filename),
        Bucket=bucket,
        Key=f'{filename}'
    )

Note: AWS Execution time limit has a maximum of 15 minutes so can you process your HUGE files in this amount of time? You can only know by testing.

The /tmp/ directory is limited to 512MB for AWS Lambda functions.

If you search StackOverflow, you'll see some code from people who have created Zip files on-the-fly without saving files to disk. It becomes pretty complicated.

An alternative would be to attach an EFS filesystem to the Lambda function. It takes a bit of effort to setup, but the cost would be practically zero if you delete the files after use and you'll have plenty of disk space so your code will be more reliable and easier to maintain.

# For me below code worked for single file in Glue job to take single .txt file form AWS S3 and make it zipped and upload back to AWS S3. 
import boto3
import zipfile
from io import BytesIO
import logging
logger = logging.getLogger()

s3_client = boto3.client('s3')
s3_resource= boto3.resource('s3')

# ZipFileStream function declaration
self._createZipFileStream(
                    bucketName="My_AWS_S3_bucket_name",
                    bucketFilePath="My_txt_object_prefix", 
                    bucketfileobject="My_txt_Object_prefix + txt_file_name",
                    zipKey="My_zip_file_prefix")

# ZipFileStream function Defination
def _createZipFileStream(self, bucketName: str, bucketFilePath: str, bucketfileobject: str, zipKey: str, ) -> None:
    try:
        obj = s3_resource.Object(bucket_name=bucketName, key=bucketfileobject)
        archive = BytesIO()

        with zipfile.ZipFile(archive, 'w', zipfile.ZIP_DEFLATED) as zip_archive:
            with zip_archive.open(zipKey, 'w') as file1:
                file1.write(obj.get()['Body'].read())  

        archive.seek(0)

        s3_client.upload_fileobj(archive, bucketName, bucketFilePath + '/' + zipKey + '.zip')
        archive.close()
            
        # If you would like to delete the .txt after zipped from AWS S3 below code will work. 
        self._delete_object(
                bucket=bucketName, key=bucketfileobject)

    except Exception as e:
        logger.error(f"Failed to zip the txt file for {bucketName}/{bucketfileobject}: str{e}")

# Delete AWS S3 funcation defination.
def _delete_object(bucket: str, key: str) -> None:
        try:
            logger.info(f"Deleting: {bucket}/{key}")
            S3.delete_object(
                Bucket=bucket,
                Key=key
            )
        except Exception as e:
            logger.error(f"Failed to delete {bucket}/{key}: str{e}")`enter code here`