I am wondering if there are any ramifications in uploading files that are roughly 4GB in size through a web app using Django/Python? I remember in the past streaming uploads using Java was the preferred method but does this still today or is it perfectly safe to do so with Django/Python?
-
"perfectly safe"? It's always been perfectly safe. What could possibly be "unsafe" about it? What are you worried about? What problems would you like to avoid?S.Lott– S.Lott2011-11-23 21:17:22 +00:00Commented Nov 23, 2011 at 21:17
-
4Java or similar (as a browser plugin -- that part is key) has been used for this type of thing in the past because it can then account for starts, stops, and resuming transfers. Uploading 4GB over HTTP is insane (simply using a standard file upload field). Period.Chris Pratt– Chris Pratt2011-11-23 21:29:45 +00:00Commented Nov 23, 2011 at 21:29
-
8Chris, Why is it insane? Can you give more details? Can you suggest an alternative given the fact that the uploaded file is relative to the web app and user account so FTP etc is not a solution.jeffci– jeffci2011-11-23 21:56:49 +00:00Commented Nov 23, 2011 at 21:56
-
Hi @tdelam - below you mention you had a solution utlising celery. Could I ask if it were possible to share this solution with me?Micheal J. Roberts– Micheal J. Roberts2020-08-12 13:28:28 +00:00Commented Aug 12, 2020 at 13:28
-
@MichealJ.Roberts Sure, this was years ago but I'll do my best to remember. I had created a celery task and used rabbit to dispatch messages. When a huge file was being uploaded, I used celery to do the uploading and encoding of the videos in the background. Once that was complete, rabbit would get a message and send the notification that their upload is complete. I did something like this: stackoverflow.com/questions/19231389/…jeffci– jeffci2020-08-12 19:44:33 +00:00Commented Aug 12, 2020 at 19:44
Add a comment
|
3 Answers
Django will by default, put uploaded file data into memory if it is less than 2.5MB. Anything larger will be written to the server's /tmp directory and then copied across when the transfer completes. Many of Django's file upload settings can be customised, details are available in the documentation. You can also customise the file handling and you'll certainly want to do this.
Before we consider any technical constraints, uploading such large files with the browser will give the user a very poor experience. There is no feedback about how the transfer is going (although google chrome does display the upload status as a percentage) and no way to pause or resume transfers.
You are also likely to run into problems on the server. Apart from the extremely long time that each thread will be taken with dealing with the streamed data, you have the time it takes for the system to copy the resulting file from /tmp to its correct location.
Unless you are very confident that you can foresee any problem that the server might have with the uploads, I would suggest that this is a bad idea. It's pretty hard to find any information on this via google and there do seem to be a lot of hits that describe problems with large file uploads.
While Django is technically capable of receiving uploaded files this large, the very poor user experience and technical difficulties mean this may not be the best approach. Have you considered using dedicated software to handle the file transfer?
3 Comments
jeffci
hi adamnfish, thanks for the thorough response. I have considered a separate service, my only issue with that is this is a specific web app for specific groups, they will be uploading full movies that they've created, which can be ~4GB so I am setting a limit to 4GB. The idea is that they will be able to upload these files and it will be tied to their account. They can have multiple files in their account. I am aware of the user experience and they are as well and they are fine with that, I just need to figure out the best way to do this so there are no server issues
juliomalegria
Use
django-chunked-uploads! (disclaimer: I wrote that library).
eisa nahardani
Hi @juliomalegria, I take a look at your library. how fast is the speed of upload ? Do i need to use
ftp yet or it is enough?For future readers:
To up the max filesize allowed with in memory storage set the following in your settings.py:
FILE_UPLOAD_MAX_MEMORY_SIZE = 5242880 # make it 5Mb instead of 2Mb
Of course this won't help you for 4Gb.
3 Comments
jeffci
Yes, it is easy to set it for sure. My original question (years ago) had more to do with possible issues of uploading very large files. I have since finished that application. The solution was using celery and rabbit mq to queue uploads. The application has been running for 8 years flawlessly uploading 100GB video files and encoding them.
Parikshit Chalke
@tdelam I'm also trying to upload file using celery and rabbit mq. But I can only upload files upto 2.5mb because above that size kombu gives me error TypeError: cannot serialize '_io.BufferedRandom' object. Can you help me out??
adomzy
Hi! Any chance you can explain how you worked this out? I'm trying to do something similar, and it would really help.
The last answer covers it. We routinely upload 2.5mb+ (but usually not 4gb)
adamnish link is correct, see this snippet (from his link to django docs) regarding writing the file to disk, instead of having it in memory first:
def handle_uploaded_file(f):
with open('some/file/name.txt', 'wb+') as destination:
for chunk in f.chunks():
destination.write(chunk)
More info on the "chunks" call: https://docs.djangoproject.com/en/dev/ref/files/uploads/#django.core.files.uploadedfile.UploadedFile.chunks
Page includes how to set "chunk" size, etc.
