How to share code in serverless with Python properly?

Question

I'm trying to find the best approach for robust serverless development with Python. The project is divided into multiple services using the serverless framework and versioned in a monorepo. Here's the structure I'd like to have:

service_1/
- serverless.yml
- handler.py
service_2/
- serverless.yml
- handler.py
shared
- module_a.py
- module_b.py

module_a and module_b comprise shared logic, which should be available for both services. So far I've found 2 approaches: wrap shared code in an installable package and inject it to services via pip or provide shared code as a layer. Both solutions have flaws, most importantly it's impossible to rapidly develop the app because any change requires pip. I've noticed this problem is solved in Node.js and there are many unanswered questions about Python.

There is the plugin - serverless-package-common, which seems to tackle this issue, however, it doesn't look like a go-first approach.

I appreciate any form of help.

I am not familiar with Node.js. How exactly is the problem solved there? Can you link some of the unanswered questions? Maybe they contain helpful information despite not having an accepted answer. — Lydia van Dyke
– Lydia van Dyke, Commented Apr 11, 2020 at 15:44
Well, I should have been more specific about the unanswered questions - I meant questions like the one I'm asking on my mind. In Node.js shared code might be resolved like in this post. — Mariano Italiano
– Mariano Italiano, Commented Apr 11, 2020 at 16:36

msc · Accepted Answer · 2020-06-19 16:23:35Z

4

Maybe you can consider to use Lambda layer. You can find a quick guide here: link.

Let me give you a simple example. This example has two serverless projects - library which has shared code and service which has service codes.

library - This project should have below file structure. You have serverless.yml in the root of this project and another folder which will be export as a module. python will be linked and let lambda find your module when you use lambda with python.

./
└ serverless.yml
└ common/
    └ python/
        └ Common.py

serverless.yml - As you can see, folder common is explicitly declared here to be exported. Beware that layer name Common is re-used in resources and serverless framework will automatically match this with resource reference.

service: library
  provider:
    name: aws
    runtime: python3.7
    stage: dev
    region: ap-northeast-1
  layers:
    Common:
      path: common
  resources:
    Outputs:
      CommonLayerExport:
        Value:
          Ref: CommonLambdaLayer
        Export:
          Name: CommonLambdaLayer

common/python/Common.py (printException is an example of shared function which will be used in another project)

import sys
import traceback
import os

def printException(exception, writer=None):
    if writer is None:
      writer = print
    top = traceback.extract_tb(sys.exc_info()[2])[-1]
    writer(", ".join([type(exception).__name__, str(sys.exc_info()[1]), os.path.basename(top[0]), str(top[1])]))

service serverless.yml - As you can see, layers should be included per each functions and you can refer to a specific layer with CloudFormation reference.

service: layer-test
  provider:
    name: aws
    region: ap-northeast-1
    runtime: python3.7
    stage: dev
  functions:
    exceptionExample:
      handler: handler.func
      layers:
        - ${cf:library-dev.CommonLayerExport}
      events:
        - http:
            path: exceptionExample
            method: get

handler.py - Now we can easily import shared module from shared layer.

import json
import Common

def func(event, context):
    try:
        1/0 # just to raise an exception and test the shared function
    except Exception as e:
        Common.printException(e)

    response = {
        "statusCode": 200,
    }
    return response

One thing you should be careful is that as lambda layer does not pack imported modules, you should import modules which is used by your layer in your services as well.

edited Jun 19, 2020 at 16:23

answered Jun 19, 2020 at 7:41

msc

4892 silver badges4 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Unbranded Manchester Over a year ago

Your answer seems helpful but please don't share links that may one day disappear. Instead share the info needed to answer the question in your answer.

msc Over a year ago

@UnbrandedManchester thanks for your comment, I just edited my answer and hope this helps.

yongsheng Over a year ago

@msc Would you mind having a look at stackoverflow.com/questions/63107494/… ? It is a follow up question to your answer.. thank you

Collectives™ on Stack Overflow

How to share code in serverless with Python properly?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related