0

I have code on aws ec2. Right now, it accepts input and output files from s3. Its an inefficient process. I have to upload input file to s3, copy s3 to ec2, run program, copy output files from ec2 to s3, then download locally.

Is there a way to run the code on ec2 and accept a local file as input and then have the output saved on my local machine?

Improve this question
1
  • 1
    What type of processing does the EC2 instance perform on the data? How long does it take to run? What libraries does it use? I wonder whether it could be replaced with an AWS Lambda function? Commented Aug 13, 2020 at 22:20

2 Answers 2

Reset to default
2

It appears that your scenario is:

  • Some software on an Amazon EC2 instance is used to process data on the local disk
  • You are manually transferring that data to/from the instance via Amazon S3

An Amazon EC2 instance is just like any other computer. It runs the same operating system and the same software as you would on a server in your company. However, it does benefit from being in the cloud in that it has easy access to other services (such as Amazon S3) and resources can be turned off to save expense.

Optimize current process

In sticking with the current process, you could improve it with some simple automation:

  • Upload your data to Amazon S3 via an AWS Command-Line Interface (CLI) command, such as: aws s3 cp file.txt s3://my-bucket/input/
  • Execute a script on the EC2 process that will:
  • Download the file, eg: aws s3 cp s3://my-bucket/input/file.txt .
  • Process the file
  • Copy the results to S3, eg: aws s3 cp file.txt s3://my-bucket/output/
  • Download the results to your own computer, eg: aws s3 cp s3://my-bucket/output/file.txt .

Use scp to copy files

Assuming that you are connect to a Linux instance, you could automate via:

  • Use scp to copy the file to the EC2 instance (which is very similar to the SSH command)
  • Use ssh with a [remote command(https://malcontentcomics.com/systemsboy/2006/07/send-remote-commands-via-ssh.html) parameter to trigger the remote process
  • Use scp to copy the file down once complete

Re-architect to use AWS Lambda

If the job that runs on the data is suitable for being run as an AWS Lambda function, then the flow would be:

  • Upload the data to Amazon S3
  • This automatically triggers the Lambda function, which processes the data and stores the result
  • Download the result from Amazon S3

Please note that an AWS Lambda function runs for a maximum of 15 minutes and has a limit of 512MB of temporary disk space. (This can be expanded by using Amazon EFS is needed.)

Something in-between

There are other ways to upload/download data, such as running a web server on the EC2 instance and interacting via a web browser, or using AWS Systems Manager Run Command to trigger the process on the EC2 instance. Such a choice would be based on how much you are permitted to modify what is running on the instance and your technical capabilities.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

This was very helpful, thank you. I will optimize the current process by adding the aws s3 cp scripts to my EC2 process. The code takes 10-15 minutes to run depending on input file size and requires an 80MB machine learning model, so this may exceed lambda capacity, what do you think?
Ah! You didn't mention that it was an ML model. It might be difficult to load those models into Lambda. You might also consider using Amazon SageMaker.
Thanks, will look into it. Question on implementing your suggestion. What's the best way to get the local file onto s3 and eventually my ec2? IIUC, you are suggesting to have 3 scripts, 1 local script to upload to s3, 1 script on ec2 to download from s3 and process data and cp output to s3, then another local script to download ouput from s3 to local. Is that right, or is there a way to do this all from the ec2?
The problem is that the Amazon EC2 instance, being a computer on the Internet, has no access to your own computer (which is presumably behind a router or firewall). You could totally avoid Amazon S3 by using scp to copy the resources to EC2. I've added this to my answer.
I have responded on that Question.
|
0

@John Rotenstein we have solved the problem of loading 60MB+ models into Lambdas by attaching AWS EFS volumes via VPC. Also solves the problem with large libs such as Tensorflow, opencv etc. Basically lambda layers almost become redundant and you can really sit back and relax, this saved us days if not weeks of tweaking, building and cherry picking library components from source allowing us to concentrate on the real problem. Beats loading from S3 everytime too. The EFS approach would require an ec2 instance obviously.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.