Creating an on-the-fly zip file from string content for AWS Lambda in Python

Question

I have a Python script that creates a Lambda script in AWS along with all the policies and triggers. I use python boto3 library for that. I create the zip file for the lambda as on-the-fly rather than uploading a static zip file from the hard drive. I use this simple code from below to create my zip file. It creates the zip file without any problems and my python code uploads this zip file as a lambda script and I can view my lambda script in the AWS without any problems. But when I run my lambda script it gives me the module not found error even though I can clearly see that both the module name and the file name does exist and is view-able.

Unable to import module 'xxxx': No module named xxxx

In the file system I double click that zip file that was created by this code and see that the content is created and everything looks normal.

If I bypass zipping on the fly and create the zip statically using WinZip and let the rest of the Python & boto3 script upload this file then it works just fine.

def CreateLambdaZip(self, fileName, fileContent):
    with zipfile.ZipFile('Lambda/' + fileName + '.zip', 'w') as myzipc:
        myzipc.writestr( fileName + '.py', fileContent)
        myzipc.close()

It kinda looks like for the zip file I'm skipping some special headers that is needed by Aws Lambda. Is there such thing? Because in the file system the zip file that is created by Python code and the other one that is created by WinZip are exactly the same. So I know there's nothing wrong with the lambda script.

Update: I'm uploading the zip file using the below code that reads the zip file which was created using the above snippet.

with open('Lambda/'+ fileName +'.zip', 'rb') as zipFile:
    func = boto3.client("Lambda").create_function(
                FunctionName=lambdaFunction,
                Runtime='python2.7',
                Role=role['Role']['Arn'],
                Handler= fileName + "." + functionName,
                Description=description,
                Timeout=10,
                MemorySize=256,
                Publish=True,
                Code={'ZipFile': zipFile.read()},
            )

When I use zipFile.read() I get 2 different headers for the same content when I zip it using WinZip and when I zip it using Python's module. Zip file that's created programmatically using Python

b'PK\x03\x04\x14\x00\x00\x00\x00\x00\xe4~\x01IO\x96J=Z\x07\x00\x00Z\x07\x00\x00\x19\x00\x00\x00schedule-ec2-snapshots.pyimport json\nimport boto3\nimport time\nfrom datetime import date, timedelta\n\nprint(\'Loading scheduled EC2 backup actions\')\n\ndef create_snapshots(event, context):\n    """\n    Lambda function that executes daily snapshots for the instances that

and zipfile created by WinZip

b'PK\x03\x04\x14\x00\x02\x00\x08\x004X\xfcH\x88\x1f\xce\xb5&\x03\x00\x00b\x07\x00\x00\x19\x00\x00\x00schedule-ec2-snapshots.py\x8dU]k\xdb@\x10|7\xf4?,\nA\x12qL\xda\x06B\r~I\x93Bh\x9b\x87&\xf4E\x15\xe1\xac[\xdb\xd7HwBw2\t\xc1\xff\xbd{+\xeb\xcb.\xb4\n\xc4\xba\xdb\xd1\xec\xce\xdc\xae\xa4\x8a\xd2T\x0e~[\xa3\'\xaa\xb9_\x1ag>\xb6\x0b\xa7\n\x9c\xac*S\x80\x14\x0e\xfd\n\xf6\x11\xbf\x9er\\b\xee\xc4dRVJ\xbb(\xfcf\x84Tz\r6\xdb\xa0\xacs\x94p\xfb\xf9\x03,E\xf6\\\x97

Try unzipping the file from the command line. What does it write? — Karen B
– Karen B, Commented Aug 1, 2016 at 2:54
how do you upload this zip archive, what/how cli command you are using ? — b.b3rn4rd
– b.b3rn4rd, Commented Aug 1, 2016 at 3:26
@b.b3rn4rd I'm not using CLI. I'm running a py script that uses boto3.client("lambda").create_function. Updated code above — Cüneyt Aliustaoğlu
– Cüneyt Aliustaoğlu, Commented Aug 1, 2016 at 3:49
@KarenB Yes this might be related to what you read from zipfile. When I try to read the zip file while debugging I see different binary contents for the same string content. Updated code above — Cüneyt Aliustaoğlu
– Cüneyt Aliustaoğlu, Commented Aug 1, 2016 at 4:13

Kyle · Accepted Answer · 2018-08-17 15:52:08Z

With the info above I was able to start the in-memory solution. The deployment of that zip file worked but I could not use the resulting function. Got error:

Unable to import module '<function-name>': No module named <function-name>

I got it to work by specifying the file permissions.

I then use the in-mem-zip to create an AWS lambda function.

Setup: file_map is a dictionary of full_path->file_bytes. files is a list of full_paths

def create_lambda_function(function_name, desc, role, handler, file_map, files)
    zip_contents = create_in_mem_zip_archive(file_map, files)

    result = lambda_code.create_function(
        FunctionName=function_name,
        Runtime="python2.7",
        Description=desc,
        Role=role,
        Handler=handler,
        Code={'ZipFile': zip_contents},
    )
    return result

def create_in_mem_zip_archive(file_map, files):
    buf = io.BytesIO()
    logger.info("Building zip file: " + str(files))
    with zipfile.ZipFile(buf, 'w', zipfile.ZIP_DEFLATED) as zfh:
        for file_name in files:
            file_blob = file_map.get(file_name)
            if file_blob is None:
                logger.error("Missing file {} from files".format(file_name))
                continue
            try:
                info = zipfile.ZipInfo(file_name)
                info.date_time = time.localtime()
                info.compress_type = zipfile.ZIP_DEFLATED
                info.external_attr = 0777 << 16L  # give full access
                # info.external_attr = 0644 << 16L  # -r-wr--r--
                # info.external_attr = 0755 << 16L  # -rwxr-xr-x
                zfh.writestr(info, file_blob)
            except Exception as ex:
                logger.info("Error reading file: " + file_name + ", error: " + ex.message)
    buf.seek(0)
    return buf.read()

user7890538 · Accepted Answer · 2017-04-19 14:52:22Z

0

I have experienced the exactly same problem you have. My solution is do NOT use on the fly zip file. Create a real zip file and add real file into it, and it just works. You can do that even in the lambda environment, by create filepath like "/tmp/yourfile.txt" you can create temp real file when lambda execute.

answered Apr 19, 2017 at 14:52

user7890538

1

Collectives™ on Stack Overflow

Creating an on-the-fly zip file from string content for AWS Lambda in Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related