AWS Lambda Python - Return BytesIO file?

Question

I'm setting up a function in AWS Lambda using python 3.7 and it won't let me return a bytes type

Please notice that this is not an issue with API Gateway, I'm invoking the lambda directly.

The error is : Runtime.MarshalError, ... is not JSON serializable

output = BytesIO()
#Code that puts an excel file into output...
return {
        'Content-Disposition': 'attachment; filename="export.xlsx"',
        'Content-Type': 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
        'body' : output.getvalue()
    }

If I do :

'body' : str(output.getvalue())

It outputs a corrupted file because it adds b'' to the string

If I do :

'body' : base64.b64encode(output.getvalue()).decode()

It also outputs a corrupted file, probably because it changes the binary representation of the file.

Maybe I need to upload to S3? But it doesn't fit in my flow, this is a one time file creation and it would stay in "S3 Limbo" until TTL

output.getvalue().decode('utf-8') will resolve the 'not JSON serializable' issue. Would that help? — Martin Valgur
– Martin Valgur, Commented Jul 24, 2019 at 12:55
@MartinValgur it doesn't, it's a binary bytes stream, it gives 'utf-8' codec can't decode byte 0xda in position 16: invalid continuation byte — Mojimi
– Mojimi, Commented Jul 24, 2019 at 12:56
Possible duplicate of How to return binary data from lambda function in AWS in Python? — Martin Valgur
– Martin Valgur, Commented Jul 24, 2019 at 13:04
@MartinValgur it's not, that question is related to api gateway, I already mentioned I'm not using it — Mojimi
– Mojimi, Commented Jul 24, 2019 at 13:10

Erik · Accepted Answer · 2020-08-10 04:11:53Z

2

It is not possible to return unencoded binary data from a direct invoked AWS Lambda function.

Per the docs:

If the handler returns objects that can't be serialized by json.dumps, the runtime returns an error.

The reason you can do this with API Gateway is because API Gateway is performing the conversion of the base64 JSON content your function returns into binary for you. (See documentation here)

I would need to know more about how you are invoking Lambda to be sure but I suspect you could implement this same base64 decode logic into your direct invoke client. Alternatively, if you wanted to keep the client as simple as possible, use S3 with a lifecycle hook to keep the bucket from filling up with temporary files.

edited Aug 10, 2020 at 4:11

answered Jan 12, 2020 at 23:30

Erik

1478 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Erik Over a year ago

Downvotes are hilarious. Sorry, but you can’t return a unencoded binary stream from a direct invoked lambda no mater how much you might dislike it.

maxwell Over a year ago

Is this answer out of date at this point? Your link now points to a page titled 'Enabling binary support via API gateway'

maxwell Over a year ago

Oh after reading further, it appears that it is only possible if you passthrough to an S3 bucket rather than using a lambda as an integration type.

Collectives™ on Stack Overflow

AWS Lambda Python - Return BytesIO file?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related