How do I cache a Python object in AWS lambda (defined outside the handler)

Question

Variables defined outside of the handler are cached - does this work for objects? I want to cache a client class but not sure how.

I need to instantiate the class from inside the handler because I need data from the event for the constructor. Can I define a var for it outside the handler and store it there like this?

from . import myclass

myobj = None

def lambda_handler(event):
    myobj = myclass.MyClass(event)

myobj is set to None because I need to set a var to something correct? It won't get reset to None every invocation, will it? I want to cache the instantiated object which takes the event in its constructor.

Marcin · Accepted Answer · 2020-05-11 01:49:22Z

6

General pattern to reuse lambda execution context is following:

myobj = None

def lambda_handler(event):

    global myobj

    if not myobj:
        myobj = myclass.MyClass(event)

So if your object is None, you will create it. Subsequent lambda invocation will likely reuse the execution context and myobj will be already set.

But the problem I see in your case is that for each invocation event maybe different. This off course will lead to issues as myobj may hold old event data from old invocation.

So if event changes with each invocation, you will have to ensure somehow that your myobj holds data associated with current invocation, not previous one.

Edit: Based on the @red888 comments.

edited May 11, 2020 at 1:49

answered May 11, 2020 at 1:24

Marcin

241k16 gold badges315 silver badges368 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

red888 Over a year ago

Awesome so glad it works! I want to use the object's state instead of setting a bunch of global vars. This is a simplified version of what im actually doing (thought it would complicate the question). Point was I need some data from the event. the specific field im pulling should not change on subsequent invocations. specifically Im storing credentials from secrets manager in the object and will have retry logic that catches auth errors and updates them in the objects state- does that make sense?

Marcin Over a year ago

@red888 Yes. That's the primary purpose of a reuse of an execution context - so that you don't reinitialized constant data all the time. Examples are DB connections, downloading some files, initialization data from external api, authentication info.

red888 Over a year ago

hmm flake8 complains about this, it says local variable 'myobj' defined in enclosing scope on line 10 referenced before assignment

Marcin Over a year ago

@red888 Maybe if you provide real code, it will be easier to see what's going on. Could be new question as it might be new issue. Does the code work in lambda itself?

red888 Over a year ago

the application works but my linter is complaining about the variable assignment: flake8(F823) flake8rules.com/rules/F823.html. Looks like it wants me define it as global myobj on a line before the if condition. that wont modify caching behavior will it? Looks like its just more explicit syntax right?

|

luketn · Accepted Answer · 2022-02-02 09:44:15Z

I've been using the following pattern, with a dictionary for the warm resources in the lamdba:

from time import time
import json

warm_resources = {
    'cold_start_time': time(),
    'invocation_count': 0
}


def lambda_handler(event, context):
    invocation_time = time()
    warm_resources['invocation_count'] = warm_resources['invocation_count'] + 1
    print(f'Invocation count: {warm_resources["invocation_count"]}, Seconds since cold start: {round(invocation_time - warm_resources["cold_start_time"], 1)}')
    return {
        'statusCode': 200,
        'body': json.dumps({
            'Howdy': 'Ho'
        })
    }

I haven't found it necessary to use the global keyword (PyCharm seems happy :)), and the dictionary of warm_resources is persistent between invocations.

Collectives™ on Stack Overflow

How do I cache a Python object in AWS lambda (defined outside the handler)

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related