Compress large string into json serializable object for AWS lambda function output

Question

I have a large string that I would like to send as the response for a python lambda function on AWS. The documentation states that the response must be json serializable (must run through json.dumps). My string is roughly 12MB which is larger than the maximum allowed payload on AWS. If I gzip the string it is compressed to roughly 2MB, but then the object is no longer json serialiable.

Here is a minimal example:

import gzip
import json
largeString=b"sdlfkjs dlfkjs dflkj "
compressed=gzip.compress(largeString)
out={}
out['data']=compressed
json.dumps(out)

which returns the expected error:

TypeError: Object of type bytes is not JSON serializable

Mark Adler · Accepted Answer · 2022-01-09 22:02:26Z

1

Use base64.b85encode(). This will expand the compressed data by 25%, so hopefully 2.5MB still fits.

Note that compressing may not be a robust solution to your problem. If your data gets larger or less compressible, you may still bust the payload limit.

answered Jan 9, 2022 at 22:02

Mark Adler

115k15 gold badges137 silver badges183 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Compress large string into json serializable object for AWS lambda function output

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related