Django-Storages with SFTP: GET-requests fail

Question

I am trying to use django-storages to access my "Hetzner" Storage Box (https://www.hetzner.com/storage/storage-box) using SFTP which should hold media data, i.e. image files which users of my website can upload dynamically.

The corresponding part of my settings.py file looks like:

DEFAULT_FILE_STORAGE = 'storages.backends.sftpstorage.SFTPStorage'
SFTP_STORAGE_HOST = 'username.your-storagebox.de'
SFTP_STORAGE_ROOT = '/media'

SFTP_STORAGE_PARAMS = {
'username': 'username',
'password': 'password',
'allow_agent': False,
'look_for_keys': False,
}

The strange thing is that, when the user uploads an Image, it is placed on the storage space as I can confirm using SFTP. But getting the images from the storage box fails, no Image is displayed. An excerpt from the console:

[03/Sep/2021 22:34:01] "GET /media/filename.jpg HTTP/1.1" 404 1962

I could figure out that Django is still looking inside my MEDIA_DIR for the files. Againg, the corresponding part of my settings:

MEDIA_DIR = 'media'
MEDIA_ROOT = os.path.join(BASE_DIR, MEDIA_DIR)
MEDIA_URL = '/media/'

So in a nutshell: Using SFTP seems to be working for putting files into storage, but getting them again somehow fails.

EDIT: As requested, I am going to provide some more code snippets: models.py:

class SizeRestrictedImageField(ImageField):

    def __init__(self, *args, **kwargs):
        self.max_upload_size = kwargs.pop('max_upload_size', 0)
        super().__init__(*args, **kwargs)

    def clean(self, *args, **kwargs):
        data = super().clean(*args, **kwargs)

        file = data.file
        try:
            if file.size > self.max_upload_size:
                raise forms.ValidationError(_('Please keep filesize under %s. Current filesize %s'
                        ) % (filesizeformat(self.max_upload_size),
                        filesizeformat(file.size)))
        except AttributeError:
            logger.exception('An Exception occured while checking for max size of image upload. size: `%s`'
                             , file.size)
            pass

        return data


class ImageModel(models.Model):
    image = SizeRestrictedImageField(upload_to=POST_PIC_FOLDER, null=True, blank=True,
                              help_text="Erlaubte Dateitypen: .jpeg, .jpg, .png, .gif", max_upload_size=MAX_IMAGE_SIZE)

And my urls.py:

urlpatterns = [
                  path('defaultsite/', defaultsite_view, name='home'),
                  path('help', help_view, name="help"),
                  path('user/', include('user.urls')),
                  path('sowi/', include('sowi.urls')),
                  path('blog/', include('blog.urls')),
                  path('chat/', include('chat.urls')),
                  path('notifications/', include('notifications.urls')),
                  path('cookies/', include('cookie_consent.urls')),
                  path('', home_view, name="home"),
                  path('about/', AboutUsView.as_view(), name="about-us"),
                  path('impressum/', impressum_view, name="imprint"),
                  path('privacy/', privacy_view, name='privacy'),
                  path('privacy/statement/', privacy_statement_view, name='privacy-statement'),
                  path('agb', agb_view, name="agb")
              ] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT) + static(settings.MEDIA_URL,
                                                                                           document_root=settings.MEDIA_ROOT)

I tried removing the +static(...)-part from my url-patterns, but that didn't seem to solve the issue.

Hi Sarius, I feel having some snippet of the code for the get would be helpful. — Paulo
– Paulo, Commented Sep 5, 2021 at 21:03
Can u please add project.urls code of URL patterns in the question? — Faisal Nazik
– Faisal Nazik, Commented Sep 5, 2021 at 21:18
@Sarius, I might have another idea based on this issue (github.com/jschneier/django-storages/issues/1045). I don't think you can access your file the same way you used to. GET /media/file.jpg is going to redirect you to the media directory indeed. You need to create a proxy to the sftp to get the file. I believe you might find the So post usefull stackoverflow.com/questions/50618855/… — Paulo
– Paulo, Commented Sep 6, 2021 at 8:35
thanks @Paulo. I will try it and if it works, i'll tell you. — Sarius
– Sarius, Commented Sep 6, 2021 at 8:47

Paulo · Accepted Answer · 2021-09-06 09:00:10Z

1

+50

Check django-storage setup

I feel you may have forgot to migrate your fields in django models ? In django-storage documentation on Github, you have those snippet of code.

From:

photo = models.FileField(
    storage=FileSystemStorage(location=settings.MEDIA_ROOT),
    upload_to='photos',
)

to:

photo = models.FileField(
    upload_to='photos',
)

Could that be it ? (as mention in the comment, having some snippet of code would greatly help.

SFTP access

Django-storage act has a proxy to save you files some place. I can be a s3 bucket, an http cdn like. Or in you case a SFTP server.

with other back-end supporting the HTTP protocol it is fairly easy to get the file back. As the back-end will provide you with a link directly to the content you stored.

For SFTP, this is going to be different, web pages does not natively support FTP protocol. So in order to access the file, you will have to create a proxy layer between your web pages, and the FTP server.

@action(methods=['get'], detail=True)
def download(self, request, pk=None):

    try:
        obj = ImageModel.objects.get(id=pk)
    except ImageModel.DoesNotExist:
        raise Http404

    # with some SFTP client
    # 1. check the file exist
    # 2. pull the file from the server
    # 3. attach it to the response with the proper header
    stream = sftp_client.open(obj.file.name)
    file = stream.read()

    type, encoding = mimetypes.guess_type(obj.file.name)
    response = HttpResponse(file, content_type=type)
    response['Content-Disposition'] = u'attachment; filename="{filename}'.format(
            filename=obj.file.name)
        return response
    raise Http404

edited Sep 6, 2021 at 9:00

answered Sep 5, 2021 at 21:08

Paulo

11k5 gold badges23 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sarius Over a year ago

Hi Paulo, thanks. I already checked that, this doesn't solve the problem.

Sarius Over a year ago

How do I create such a proxy layer?

Sarius · Accepted Answer · 2021-09-07 10:35:42Z

1

I want to complete @Paulos Answer. You can create the proxy by using a middleware. Create a file project/middleware.py and add it to your middleware-array in the settings.py.

Then create the middleware:

import mimetypes

from storages.backends.sftpstorage import SFTPStorage
from django.http import HttpResponse



class SFTPMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
        # One-time configuration and initialization.

    def __call__(self, request):
        # Code to be executed for each request before
        # the view (and later middleware) are called.

        SFS = SFTPStorage()
        
        response = self.get_response(request)

        path = request.get_full_path()

        if SFS.exists(path):
            file = SFS._read(path)
            type, encoding = mimetypes.guess_type(path)
            response = HttpResponse(file, content_type=type)
            response['Content-Disposition'] = u'attachment; filename="{filename}"'.format(filename=path)

        return response

EDIT: It is actually not necessary to open up a new connection on every call. You should rather create a single connection on system startup in the ready()-hook of your config.py and use this one.

edited Sep 7, 2021 at 10:35

answered Sep 6, 2021 at 11:21

Sarius

1553 silver badges19 bronze badges

1 Comment

BlindSide Over a year ago

Hi Sarius, i do have a question to your solution. I saved the middleware.py into my Django_project folder. And added the file to Middleware Array. ` "django.middleware.common.CommonMiddleware", "django_project.middleware.SFTPMiddleware", "django.middleware.csrf.CsrfViewMiddleware",` The uploaded image is still an /media/ on PC. Do I have something else to do? Thanks

Collectives™ on Stack Overflow

Django-Storages with SFTP: GET-requests fail

2 Answers 2

Check django-storage setup

SFTP access

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Check django-storage setup

SFTP access

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related