5

What I would like to do:

What I would like to do is have a url which would return to the caller a CSV file which is essentially a export of data. I would like this to remain to be a serverless solution.

What I have done:

I have created an AWS API Gateway with the URL I want. I have created a lambda that will query the database and create a CSV string of that data. That data is placed in a JSON object and returned. API gateway then gets the CSV data from the json object and returns CSV to the caller with appropriate headers to indicate tht it is a CSV and attachment. Testing from the browser I get the download automatically just like I intended.

The problem I see:

This works well until there is a sizable amount of data at which point I start getting "body size is too long".

My attempts to resolve:

I did some googling around and I see others have had similar issues. In one solution I saw that they return a link to the file that they created. This solution seems viable for them because they had a server. For my serverless architecture it seems to be a little trickier. I could take and store the file into S3 but then i would have to return a link to S3. That seems like it could work but doesn't feel right like im missing a configuration option. It also feels like im exposing the implementation by returning the s3 urls as well.

I have looked around for tutorials and example of people doing similar things and i haven't found any.

My Questions:

Is there a way to do this? Is there another solution that i dont know of? How do i return a file, in this case CSV, from API Gateway of a larger size

Improve this question
1
  • I'm stuck on just getting the CSV to generate and start downloading. How did you set that up? Commented Nov 14, 2018 at 21:05

1 Answer 1

Reset to default
15

There is a limit of 6 MB for AWS Lambda response payloads. If the files you need to server are larger than that you won't be able to serve them directly from Lambda.

Using S3 to store and serve the files is the standard way of doing something like this. I would leave the S3 bucket private and generate S3 Pre-signed URLs in the Lambda function. That will limit the time that the CSV file is available for download, and it will prevent people from being able to guess the URLs of files you are serving. You would use an S3 Lifecycle Policy to archive or delete the files after a period of time.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thank you. I was just exploring this idea but didn't know of the s3 pre-signed url. Im going to look into that now because that seems to solve some of my internal concerns
@Nexeh you can also place CloudFront in front of your S3 bucket which will further hide your implementation details from your end users. It will also speed up the file downloads. docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/…
I wrote the file to S3, generated a pre-signed url, returned it to the call through api gateway and used my angular client to download the file using the provided url. Thank you so much! I feel more confident in the solution now.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.