AWS Lambda post-publish script?

Question

We're trying to use ffmpeg via a function on AWS Lambda. We include ffmpeg and ffprobe in our include list so that the files are available when published, but we have to move them to somewhere inside the /tmp directory so that we can make them executable.

Currently we have code which will copy them to this folder, and then run chmod -R +x /tmp/ffmpeg to make the files in the folder executable. From there on it's fine, but it seems a bit excessive to have to perform this the first time the function is executed.

We've looked at the "scripts" option of the package.json file, and tried the following (amongst other things), but nothing seems to make a difference.

"scripts": {
    "postpublish": [
        "mkdir /tmp/ffmpeg",
    ]
}

I see that Amazon have suggested that people set permissions in the .zip file they upload so that the executables have the execute permission without being copied to a writeable folder and chmod'd, but we have two issues with this:

We're using the Visual Studio tools from Amazon to publish.
Only one proprietary format of zip (Info-Zip) seems to support permissions in zip files, and the last stable release of their software was 8 years ago.

Is there any way we can run some kind of post-publish script to copy and set permissions on the ffmpeg libraries?

Update: It seems that MacOS can generate a zip file with the permissions of the two executables, which is then honoured by Lambda. I'm still not sure how we can achieve this from Windows.

Does not Elastic Transcoder solve your problem? aws.amazon.com/elastictranscoder — Cagatay Gurturk
– Cagatay Gurturk, Commented Mar 8, 2017 at 10:41
@ÇağatayGürtürk Not if you want to extract thumbnails from specific timestamps. We're actually using ZenCoder for video transcoding, but we use ffmpeg to generate previews in response to user changes. Elastic Transcoder only supports intervals. Thanks anyway. — ProgrammingLlama
– ProgrammingLlama, Commented Mar 8, 2017 at 10:54
Not sure why you think "it seems a bit excessive to have to perform this the first time the function is executed". The Lambda function is not deployed to a container until the first time it is executed, so it seems perfectly reasonable to me to perform this step on initial execution. It may also be deployed to other containers based on your usage pattern, at which point the first execution in a new container will need to perform this file copy task again. — Mark B
– Mark B, Commented Mar 8, 2017 at 15:53
@MarkB Well, uploading a zip file created on MacOS preserves the "execute" permission of the ffmpeg executable, but on Windows it doesn't seem to. It seems excessive because it's a step we don't need to perform if we deploy from MacOS. — ProgrammingLlama
– ProgrammingLlama, Commented Mar 9, 2017 at 0:51

ProgrammingLlama · Accepted Answer · 2017-03-09 03:19:53Z

This is a long way from the desired right-click "Publish to AWS Lambda" option in Visual Studio, but it gets the job done.

For some unknown reason, Amazon require that the ZIP file contain the Unix-style file permissions. As a Windows user, I was like "ZIP files? With permissions? Huh?" because Windows ZIP files don't really do permissions.

After some searching, I found a Python script that is capable of setting the permissions in a ZIP file. Awesome. So I tried it. It didn't work. It turns out that, as part of the file metadata, ZIP file store the "host OS" that the file was created on. 7zip helpfully told me that my files were "FAT".

So, here is my solution (which can clearly be improved and probably automated):

import zipfile
import os

srcFileName = os.getcwd() + '\\bin\\Release\\netcoreapp1.0\\prepublish.zip'
tmpFileName = os.getcwd() + '\\bin\\Release\\netcoreapp1.0\\publish.zip'

with zipfile.ZipFile(srcFileName, 'r') as sz:
    with zipfile.ZipFile(tmpFileName, 'w', zipfile.ZIP_DEFLATED) as z:
        for name in sz.namelist():
            srcFile = sz.open(name, "r")
            print(name)
            zi = zipfile.ZipInfo(name)
            zi.create_system = 3
            zi.external_attr = (0o764 << 16)
            z.writestr(zi, sz.read(name))
        z.close()
        sz.close()

It takes an existing zip file (prepublish.zip) which contains my code, and re-zips it as publish.zip, but with 764 file permissions and a "Unix" host OS.

I hope this helps someone.

Collectives™ on Stack Overflow

AWS Lambda post-publish script?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related