2

I have a requirement to upload a CSV file using Rest API, process this file in AWS Lambda, then provide a response in API back to the user. How can I achieve this? I thought of uploading the file to s3 using API and then triggering a lambda function based on s3 events, but in this case, I won't be able to provide a response to the user.

Improve this question
7
  • What type of response are you looking for when the user upload the file? Commented Jan 25, 2023 at 6:14
  • @AlbertMarrero after processing output will be generated in form of new csv and that will be returned to user in API response. Commented Jan 25, 2023 at 8:02
  • Do you have a limit on how big the file uploaded can be? I feel that you are doing too much in one call, and maybe you need to break up. I am concerned that you might reach a Lambda timeout if you do too much. Commented Jan 25, 2023 at 8:36
  • 1
    @AlbertMarrero file will be max 3 MB. how can I break this into multiple steps? I was thinking of creating PUT API to upload the file, then creating a trigger to invoke lambda, process the file, and upload the file to s3. problem is, how can I now send the output file link to user? Commented Jan 25, 2023 at 9:24
  • For the output file link, you can return a pre-signed URL. Commented Jan 26, 2023 at 7:14

1 Answer 1

Reset to default
1

I am adapting this article to your question.

Create a PUT method for your API for uploading CSV file

  1. Choose Actions, and then choose Create Method.
  2. From the dropdown list, choose PUT and then choose the check mark icon.
  3. Under the Integration type category, choose AWS Service.
  4. For AWS Region, choose us-east-1 or the AWS Region you see on the Bucket properties page.
  5. For AWS Service, choose Simple Storage Service (S3).
  6. Leave AWS Subdomain empty.
  7. For the HTTP method, choose PUT.
  8. For Action Type, choose Use path override.
  9. For Path override (optional), enter {bucket}/{key}.
  10. For the Execution role, enter the Amazon Resource Name (ARN) for the IAM policy for the API Gateway role.
  11. For Content Handling, choose Passthrough.
  12. Choose Save.

Configure parameter mappings for the PUT method

  1. In the Resources panel of your API page, choose PUT.
  2. Choose Integration Request.
  3. Expand URL Path Parameters.
  4. Choose Add path.
  5. For Name, enter bucket.
  6. For Mapped from, enter Method.request.path.folder.
  7. Choose the checkmark icon at the end of the row.
  8. Repeat steps 4 through 7. In step 5, set Name to key. Step 6 sets Mapped from to Method.request.path.object.

Create a GET method for your API for retrieving a CSV file

  1. In the Resources panel of your API page, choose /{object}.
  2. Choose Actions, and then choose Create Method.
  3. From the dropdown list, choose GET and then choose the check mark icon.
  4. Under the Integration type category, choose AWS Service.
  5. For AWS Region, choose us-east-1, or the Region you see on the Bucket properties page.
  6. For AWS Service, choose Simple Storage Service (S3).
  7. Leave AWS Subdomain empty.
  8. For the HTTP method, choose GET.
  9. For Action Type, choose Use path override.
  10. For Path override (optional), enter {bucket}/{key}.
  11. For the Execution role, enter the Amazon Resource Name (ARN) for the IAM policy for the API Gateway role.
  12. For Content Handling, choose Passthrough.
  13. Choose Save.

Configure parameter mappings for the GET method

  1. In the Resources panel of your API page, choose GET.
  2. Choose Integration Request.
  3. Expand URL Path Parameters.
  4. Choose Add path.
  5. For Name, enter bucket.
  6. For Mapped from, enter Method.request.path.folder.
  7. Choose the checkmark icon at the end of the row.
  8. Repeat steps 4 through 7. In step 5, set Name to key. Step 6 sets Mapped from to Method.request.path.object.

Set up response mapping to see the CSV in the browser

  1. In the Resources panel of your API page, choose GET.
  2. Choose Method Response.
  3. Expand 200.
  4. Under Response Body for 200, remove application/JSON.
  5. Under Response headers for 200, choose Add header.
  6. For Name, enter content type.
  7. Choose the check mark icon to save.
  8. Choose Method execution to go back to the Method Execution pane.
  9. Choose Integration Response.
  10. Expand 200, and then expand Header Mappings.
  11. Choose the pencil icon at the end of the row named content-type.
  12. Enter text/CSV to see a CSV file.

Set up binary media types for the API

  1. In the navigation pane of your API page, choose Settings.
  2. In the Binary Media Types section, choose Add Binary Media Type.
  3. In the text box, add the following string: /

Note: Avoid putting the string in quotes. You can substitute a wildcard for a particular Multipurpose Internet Mail Extensions (MIME) type you want to treat as a binary media type. For example, choose "image/jpeg" to have API Gateway treat JPEG images as binary media types. If you add /, then API Gateway treats all media types as binary media types.

  1. Choose Save Changes.

Resolve CORS error with binary settings for API

  1. If you want to use the aforementioned APIs (PUT and GET) in a web application, you might encounter a CORS error.
  2. To resolve the CORS error with binary settings turned on, start CORS from the API Gateway console.
  3. In the Resources panel of your API page, select /{object}.
  4. For Actions, choose Enable CORS.
  5. Choose Enable CORS and replace existing CORS headers.
  6. Use the update integration CLI command to update the content handling property. This update allows the binary content to be handled for preflight options requests with mock integration.
  7. Run the two CLI commands that follow by updating the API ID, Resource ID, and AWS Region. You can obtain the API ID and Resource ID from the top of the Gateway API console when you select the {object} resource.

Finally, deploy your API and upload a CSV file via your API

curl -i --location --request PUT 'https://abc12345.execute-api.us-west-2.amazonaws.com/v1/testfolder/test.csv' --header 'Content-Type: text/csv' --data-binary '@/Path/to/file/test.csv'
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.