aws s3 upload optimization，using java sdk

Question

Now, I have a server to accept media stream and record to mp4 files, and upload to s3, when the concurrency level is high, I found that it is the s3 uploading which cost the most cpu, and when I monitored the IO, I also found that the cpu cost changed along with the IO out to s3, and the IO(out) rate is not stable.

so I am wondering that is there exit some method that I can use to control the upload speed to s3? For example, when this second the IO rate is too much high, then I make it slow down, so it can divide the total files among every second. And I an also wondering that if it is workable to decline the cpu cost.

I wish someone can help me out, thanks and forgive me for my poor English.

otakumike · Accepted Answer · 2016-12-02 21:16:26Z

1

AWS S3 doesn't provide any solution for slowing down transfer that I am aware of, however, you could try using multipart uploads (中文) to break your upload into multiple concurrent streams. This might make your upload more manageable.

Note that you cannot use this technique for files smaller than 5MB as each upload part must be at least 5MB (except the last part). Thanks to @michael - sqlbot for correcting that)

(你的英文很好，我可以懂你)

edited Dec 2, 2016 at 21:16

answered Dec 2, 2016 at 10:19

otakumike

1758 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Michael - sqlbot Over a year ago

You can't use the technique for files < 5MB. It works, but will only result in more overhead because a file < 5MB can be uploaded only in one part. Where did you see 100 KB?

otakumike Over a year ago

You're right. I had my wires crossed somehow. I updated the answer. Thank you!

wanghao qin Over a year ago

Thanks, I had tried it before, but by this method I just found I can't be awared that when the uploads finished(I use S3ProgressListener to be awared that whether the uploads are finished, but when I use multipart uploads, I found that the transfer event changes to TRANSFER_COMPLETED_EVENT when every segment transfered complete, and I am not sure the total number of one file), by the way, how can you know I am a Chinese(I guess by my name ^_^..)?

Collectives™ on Stack Overflow

aws s3 upload optimization，using java sdk

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related