Cleaning up after AWS Lambda execution context is closed with Python

Question

From the Best Practices for Working with AWS Lambda Functions:

Take advantage of execution context reuse to improve the performance of your function. Initialize SDK clients and database connections outside of the function handler, [...]

I would like to implement this principle to improve my lambda function, where a database handle is initialized and closed every time the function is invocated. Take the following example:

def lambda_handler(event, context):
    # Open a connection to the database
    db_handle = connect_database()    

    # Do something with the database
    result = perform_actions(db_handle)  

    # Clean up, close the connection
    db_handle.close()       

    # Return the result
    return result

From my understanding of the AWS documentation, the code should be optimized as follows:

# Initialize the database connection outside the handler
db_handle = conn_database()

def lambda_handler(event, context):
    # Do something with the database and return the result
    return perform_actions(db_handle)

This would result in the db_handle.close() method not being called, thus potentially leaking a connection.

How should I handle the cleanup of such resources when using AWS Lambda with Python?

Tuan Vo · Accepted Answer · 2020-03-07 14:36:41Z

1

+50

Many people looking for the same thing with you. I believe it is impossible at this time. But we could handle the issue from the database side.

Take a look at this one

answered Mar 7, 2020 at 14:36

Tuan Vo

2,09513 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Ionut Ticus · Accepted Answer · 2020-03-07 14:54:25Z

0

The connection leak would only happen while the Lambda execution environment is alive; in other words the connection would timeout (be closed) after the execution environment is destroyed.

Whether a global connection object is worth implementing depends on your particular use case:
- how much of the total execution time is taken by the database initialization
- how often your function is called
- how do you handle database connection errors

If you want to have a bit more control of the connection you can try this approach which recycles the database connection every two hours or when encountering a database-related exception:

# Initialize the global object to hold database connection and timestamp
db_conn = {
    "db_handle": None,
    "init_dt": None
}

def lambda_handler(event, context):
    # check database connection
    if not db_conn["db_handle"]:
        db_conn["db_handle"] = connect_database()
        db_conn["init_dt"] = datetime.datetime.now() 
    # Do something with the database and return the result
    try:
        result = do_work(db_conn["db_handle"])
    except DBError:
         try:
             db_conn["db_handle"].close()
         except:
             pass
         db_conn["db_handle"] = None
         return "db error occured"      
    # check connection age
    if datetime.datetime.now() - db_conn["init_dt"] > datetime.timedelta(hours=2):
         db_conn["db_handle"].close()
         db_conn["db_handle"] = None
    return result

Please note I haven't tested the above on Lambda so you need to check it with your setup.

answered Mar 7, 2020 at 14:54

Ionut Ticus

2,7992 gold badges23 silver badges28 bronze badges

3 Comments

Kurt Stolle Over a year ago

The connection would indeed timeout after a while (depending on the database settings), but that's not the same as cleaning up resources. My question is specifically about cleaning up resources (of any kind) before the execution context is cancelled.

Ionut Ticus Over a year ago

I don't think there is a Lambda execution environment destroyed event hook at the moment; you can try to reduce the leak duration by adjusting the recycle interval and idle connection timeout setting for the database (idle_in_transaction_session_timeout for PostgreSQL, wait_timeout for MySQL etc.). You can also put a connection pooler in between.

Kurt Stolle Over a year ago

This solves the issue in the example, but doesn't answer the question

Collectives™ on Stack Overflow

Cleaning up after AWS Lambda execution context is closed with Python

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related