AWS Lambda start an async process with Python (return without waiting)

Question

I should start by saying I really know nothing about AWS, Lambda, or for that matter much about Python, so the likelihood of dumb mistakes is high.

I'm trying to create a system wherein a user POSTS a thing, the handler quickly passes them back a jobID, and then the user can revisit later to check on the process/get the result.

Basically:

def long_function(jobID, stuff): //dont know how to define this to be called asychronously
  //slow-running code
  //stuff the result in dynamoDB
  return

def lambda_handler(event, context):
    jobID = uuid.uuid4.hex
    stuff = event['stuff']
    long_function(jobID, stuff) //dont know how to write this call
    return jobID //should run before long_function completes

My understanding is that general python async stuff (which I'm also unfamiliar with) won't work, and I need to do a particular Lambda invocation. I'm failing to find a clear and simple way to do this (especially with python syntax)

Invoking a 2nd Lambda to do the real work might be a good option, and these 2 Lambdas can store the progress for a given Job ID in DynamoDB so you can implement a Lambda-based API to query the current status for Job X. Or your initial Lambda could trigger Step Functions to do the real work. — jarmod
– jarmod, Commented Aug 26, 2021 at 15:46

P. Greene · Accepted Answer · 2021-08-26 15:51:44Z

2

You are correct, the lambda function, once it returns, is done, and there is no way around that.

What you can do, also as you predicted, is launch another lambda function from within the first function (or a Batch job, or make a call to an EC2 instance for that matter).

You will need boto3 (it comes by default in all lambda Python environments) and you will need to give the lambda function the necessary IAM permissions to invoke another lambda function.

You can call either the original lambda function or another function. The code below assumes it is calling the same function.

The code itself is fairly simple:

import json
import boto3

def long_function(jobID, stuff):
    """Do something that takes a long time."""
    ...

def lambda_handler(event, context):
    # Unpack the event
    stuff = event["stuff"]
    is_long_func = event.get("is_long_func", False)

    # Route to the desired action.
    if not is_long_func:
        # Create a new job ID
        jobID = uuid.uuid4.hex

        # This is where the real business happens: invoke this
        # same lambda function with a different event.
        lam = boto3.client('lambda')
        lam.invoke_async(
            FunctionName="this-lambda-function-name",
            InvokeArgs=json.dumps({
                'jobID': jobID,
                'stuff': stuff,
                'is_long_func': True
            }).encode('utf-8')
        )
        return jobID

    else:
        # Run the long function.
        jobID = event["jobID"]
        long_function(jobID, stuff)
        return "OK"

If you need the result of the function, you can save the result of the function to s3 or similar and retrieve the contents with additional calls to the lambda function.

answered Aug 26, 2021 at 15:51

P. Greene

361 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ksyrium Over a year ago

This is a good answer, but invoke_async is now deprecated. As @Mark B points out in the other answer, one can use the InvocationType 'Event' when using invoke.

Mark B · Accepted Answer · 2021-08-26 15:39:03Z

1

It's not going to be possible for a single Lambda invocation to generate the job ID and return it, and also keep processing afterwards.

You will need something like one Lambda function that generates a job ID, invokes another Lambda function asynchronously (using the event invocation type so it won't wait for that 2nd Lambda function to finish) and then returns the job ID.

answered Aug 26, 2021 at 15:39

Mark B

203k27 gold badges339 silver badges332 bronze badges

Collectives™ on Stack Overflow

AWS Lambda start an async process with Python (return without waiting)

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related