How do I upload a file into an Amazon S3 signed url with python?

Question

I have managed to generate an Amazon S3 Signed URL with boto3.

import logging
import boto3
from botocore.exceptions import ClientError

def create_presigned_url(bucket_name, object_name, expiration=3600):
    """Generate a presigned URL to share an S3 object

    :param bucket_name: string
    :param object_name: string
    :param expiration: Time in seconds for the presigned URL to remain valid - 1 hour in this case
    :return: Presigned URL as string. If error, returns None.
    """

    # Generate a presigned URL for the S3 object
    s3_client = boto3.client('s3')
    try:
        response = s3_client.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': object_name}, ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL
    return response

result = create_presigned_url("bucket123", "test.txt")

assuming bucket123 exists. This code returns me a URL like this:

https://bucket123.s3.amazonaws.com/test.txt?AWSAccessKeyId=AKIA56EZ34KPFKO5366J&Signature=7aAZXXDwtvKGLaeu1ATHRJuaFCs%3D&Expires=1635117707

My question is:

Using python and boto3, is it possible to upload a file in this pre-signed url? How do I go on about this?

If you have boto3 then you would not use a pre-signed URL to upload a file to S3. You would simply use the boto3 methods, such as put_object and upload_file. — jarmod
– jarmod, Commented Oct 25, 2021 at 2:03
@jarmod While I agree with you, ... I need to be able to provide a pre-signed URL in this case. How would you suggest I get a URL that looks like this back? https://bekim-cribl1.s3.amazonaws.com/test.tgz?AWSAccessKeyId=AKIA56EZ34KPFKO5367J&Signature=MUUJWwA%2B9V5d7H7DjeorLYsSguo%3d&Expires=1635111412 — David
– David, Commented Oct 26, 2021 at 6:14
@Marcin Thank you so much for the help with this .. Ideally I would like to have the URL generated back in the fashion of : https://bekim-cribl1.s3.amazonaws.com/test.tgz?AWSAccessKeyId=AKIA56EZ34KPFKO5367J&Signature=MUUJWwA%2B9V5d7H7DjeorLYsSguo%3d&Expires=1635111412 Any idea on how can I achieve that? — David
– David, Commented Oct 26, 2021 at 6:15
Why do you want the pre-signed URL to look a certain way (which appears to be sigv2)? The currently-supported signing method is signature v4. Earlier signing methods are in an extended period of deprecation. — jarmod
– jarmod, Commented Oct 26, 2021 at 9:34
you can specifiy a configuration for the client. Below, it explicitly sets the signature version to 's3v4'. This version of the signature is used in the AWS Signature Version 4 signing process, which is the latest signing version for S3. self.s3_client = boto3.client('s3', aws_access_key_id=key_id, aws_secret_access_key=key_secret, config=Config(signature_version='s3v4'), region_name='eu-west-2', endpoint_url='s3.eu-west-2.amazonaws.com') — Abdullah
– Abdullah, Commented Jan 10, 2024 at 6:12

Marcin · Accepted Answer · 2021-10-24 23:53:58Z

7

Once you have the pre-signed url, you don't need boto3 at all. Instead you can use regular python functions for uploading files to S3. For example using python's requests. Also you should be using generate_presigned_post


def create_presigned_post(bucket_name, object_name,
                          fields=None, conditions=None, expiration=3600):
    """Generate a presigned URL S3 POST request to upload a file

    :param bucket_name: string
    :param object_name: string
    :param fields: Dictionary of prefilled form fields
    :param conditions: List of conditions to include in the policy
    :param expiration: Time in seconds for the presigned URL to remain valid
    :return: Dictionary with the following keys:
        url: URL to post to
        fields: Dictionary of form fields and values to submit with the POST
    :return: None if error.
    """

    # Generate a presigned S3 POST URL
    s3_client = boto3.client('s3')
    try:
        response = s3_client.generate_presigned_post(bucket_name,
                                                     object_name,
                                                     Fields=fields,
                                                     Conditions=conditions,
                                                     ExpiresIn=expiration)
    except ClientError as e:
        logging.error(e)
        return None

    # The response contains the presigned URL and required fields
    return response

result = create_presigned_post("bucket123", "test.txt")



import requests    # To install: pip install requests

# Generate a presigned S3 POST URL
object_name = 'test.txt'
result = create_presigned_post("marcin3", object_name)

# Demonstrate how another Python program can use the presigned URL to upload a file
with open(object_name, 'rb') as f:
    files = {'file': (object_name, f)}
    http_response = requests.post(result['url'], data=result['fields'], files=files)
# If successful, returns HTTP status code 204
logging.info(f'File upload HTTP status code: {http_response.status_code}')

edited Oct 24, 2021 at 23:53

answered Oct 24, 2021 at 22:32

Marcin

241k16 gold badges315 silver badges368 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

David Over a year ago

Gettting error:

<?xml version="1.0" encoding="UTF-8"?> <Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>AKIA56EZ34KPFKO53663</AWSAccessKeyId><StringToSign>POST  multipart/form-data; boundary=f9cd89cc9f8848cb3c648bb60896faae 1635120065 /bucket123/test.txt</StringToSign><SignatureProvided>tCJ1oWjPRcRs3vaP64KzxFgn1z4=</SignatureProvided>

@Marcin

David Over a year ago

error continued:

<StringToSignBytes>50 4f 53 54 0a 0a 6d 75 6c 74 69 70 61 72 74 2f 66 6f 72 6d 2d 64 61 74 61 3b 20 62 6f 75 6e 64 61 72 79 3d 66 39 63 64 38 39 63 63 39 66 38 38 34 38 63 62 33 63 36 34 38 62 62 36 30 38 39 36 66 61 61 65 0a 31 36 33 35 31 32 30 30 36 35 0a 2f 62 65 6b 69 6d 2d 63 72 69 62 6c 31 2f 74 65 73 74 2e 74 78 74</StringToSignBytes><RequestId>3D8J1523P6FS7TBV</RequestId><HostId>tnERzMP+wJsbPQcSpxiVhhdWuXFSmWkZDK+qkBKycd3ilLFHJa5FrrwBuwhfe7lz09e3vgX+Tkm=</HostId></Error>

Marcin Over a year ago

@Becks How did it go? The error persists?

David Over a year ago

The file finally uploads fine with the create_presigned_post() method. Two final remarks. I don't see the pre-signed URL printed anymore AND When I go to access the file in the S3 console, I get

<Error> <Code>AccessDenied</Code> <Message>Access Denied</Message> <RequestId>CQNCGDS4YKCPDWQ6</RequestId> <HostId> LsJI1jBMUCeQpL4qjfqipmvYeCJ5TaS3ZoBwbmXMbbklWo5stWMdVtGI50Ib+XDciHerIm5AADw= </HostId> </Error>

David Over a year ago

Accepted!!! - Thank you !!!

|

Collectives™ on Stack Overflow

How do I upload a file into an Amazon S3 signed url with python?

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related