1

I'm using django-storages to store media files in an S3 bucket. However, I am occasionally converting or otherwise fiddling with the file to create new files, and this fiddling has to use files actually on my server (most of the conversion happens using process calls). When done I'd like to save the files back to S3.

In an ideal world, I would not have to do any changes to my functions when moving from local to S3. However, I'm unsure how I would do this considering that I have to create these intermediate local files to fiddle with, and then at the end know that the resultant file (which would also be stored on the local machine) needs to then be copied over to S3.

Best that I can come up with using a pair of context guards, one for the source file and one for the destination file. The source file one would create a temporary file that would get the contents of the source file copied over, and then it would be used, manipulated, etc. The destination file context guard would just get the final desired destination path on S3 and create a temporary local file, then when exiting would create a key in the S3 bucket, copy over the contents of the temporary file, and delete it.

But this seems pretty complicated to me. It also requires me to wrap every single function that manipulates these files in two "with" clauses.

The only other solution I can think of is switching over to utilities that only deal with file-like objects rather than filenames, but this means I can't do subprocess calls.

Improve this question

1 Answer 1

Reset to default
1

Take a look at the built-in file storage API - this is exactly the use-case for it.

If you're using django-storages and uploading to S3, then you should have a line in your settings module that looks like

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'

When you're developing locally and don't want to upload your media files to S3, in your local settings module, just leave it out so it defaults to django.core.files.storage.FileSystemStorage.

In your application code, for the media files that will get saved to S3 when you move from local development to staging, instantiate a Storage object using the class returned from the get_storage_class function, and use this object to manipulate the file. For the temporary local files you're "fiddling" with, don't use this Storage object (i.e. use Python's built-in file-handling functions) unless it's a file you're going to want to save to S3.

When you're ready to start saving stuff on S3, all you have to do is set DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' again, and your code will work without any other tweaks. When that setting is not set, those media files will get saved to the local filesystem under MEDIA_ROOT, again without any need to change your application logic.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I guess my question wasn't clear. I have for example PDF files stored in S3 using the storage API. Then I am using the command line to split them into individual jpegs in the command line, and then saving the resulting files back to a folder on S3. This is all happening live, not as part of staging or development. I was able to figure it out by creating an intermediate class that creates temporary files on the local system, and then uses the storage API to save them back to S3.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.