upload file on S3 using shell script

Question

i'm trying to upload a file on my s3 but i'm getting an error concerning my signature ( SignatureDoesNotMatch ) can someone tell me what did i do wrong ? i'm using the AWS Signature Version 4 The script is taken from : https://gist.github.com/tabolario/93f24c6feefe353e14bd

the shell script :

file="/home/centos/logs/terraform-boot.log"
file_name="terraform-boot.log"
bucket="test-bucket.com"
prefix=""
region="eu-west-1"
timestamp=$(/usr/bin/date -u "+%Y-%m-%d %H:%M:%S")
signed_headers="date;host;x-amz-acl;x-amz-content-sha256;x-amz-date"
s3Key=XXXX
s3Secret=XXXX

if [[ $(uname) == "Darwin" ]]; then
  iso_timestamp=$(/usr/bin/date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%Y%m%dT%H%M%SZ")
  date_scope=$(/usr/bin/date -ujf "%Y-%m-%d %H:%M:%S"  ${timestamp}""+%Y%m%d")
  date_header=$(/usr/bin/date -ujf "%Y-%m-%d %H:%M:%S" "${timestamp}" "+%a, %d %h %Y %T %Z")
else
  iso_timestamp=$(/usr/bin/date -ud "${timestamp}" "+%Y%m%dT%H%M%SZ")
  date_scope=$(/usr/bin/date -ud "${timestamp}" "+%Y%m%d")
  date_header=$(/usr/bin/date -ud "${timestamp}" "+%a, %d %h %Y %T %Z")
fi

payload_hash() {
  local output=$(shasum -ba 256 "$file")
  echo "${output%% *}"
}

canonical_request() {
  echo "PUT"
  echo "/${prefix}/${file}"
  echo ""
  echo "date:${date_header}"
  echo "host:${bucket}.s3.amazonaws.com"
  echo "x-amz-acl:public-read"
  echo "x-amz-content-sha256:$(payload_hash)"
  echo "x-amz-date:${iso_timestamp}"
  echo ""
  echo "${signed_headers}"
  printf "$(payload_hash)"
}

canonical_request_hash() {
  local output=$(canonical_request | shasum -a 256)
  echo "${output%% *}"
}

string_to_sign() {
  echo "AWS4-HMAC-SHA256"
  echo "${iso_timestamp}"
  echo "${date_scope}/${region}/s3/aws4_request"
  printf "$(canonical_request_hash)"
}

signature_key() {
  local secret=$(printf "AWS4${s3Secret}" | hex_key)
  local date_key=$(printf ${date_scope} | hmac_sha256 "${secret}" | hex_key)
  local region_key=$(printf ${region} | hmac_sha256 "${date_key}" | hex_key)
  local service_key=$(printf "s3" | hmac_sha256 "${region_key}" | hex_key)
  printf "aws4_request" | hmac_sha256 "${service_key}" | hex_key
}

hex_key() {
  hexdump -ve '1/1 "%.2x"'; echo
}

hmac_sha256() {
  local hexkey=$1
  openssl dgst -binary -sha256 -mac HMAC -macopt hexkey:${hexkey}
}

signature() {
  string_to_sign | hmac_sha256 $(signature_key) | hex_key | sed "s/^.* //"
}

curl \
  -T "${file}" \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=${s3Key}/${date_scope}/${region}/s3/aws4_request,SignedHeaders=${signed_headers},Signature=$(signature)" \
  -H "Date: ${date_header}" \
  -H "x-amz-acl: public-read" \
  -H "x-amz-content-sha256: $(payload_hash)" \
  -H "x-amz-date: ${iso_timestamp}" \
  "https://s3-${region}.amazonaws.com/${bucket}/${file_name}"

And here is the error it retunrs :

 sh test.sh
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>XXXX</AWSAccessKeyId><StringToSign>AWS4-HMAC-SHA256
20170803T084256Z
20170803/eu-west-/s3/aws4_request
cb3baa96f5ef0417a47d49ba6a6a673ef2ff37f17b94b92b1c4e1801a2536272</StringToSign><SignatureProvided>467625e0d0fe66360068b898dd25bd9267ccabb8da71e731633a28b91cde182d</SignatureProvided><StringToSignBytes>41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 31 37 30 38 30 33 54 30 38 34 32 35 36 5a 0a 32 30 31 37 30 38 30 33 2f 65 75 2d 77 65 73 74 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 63 62 33 62 61 61 39 36 66 35 65 66 30 34 31 37 61 34 37 64 34 39 62 61 36 61 36 61 36 37 33 65 66 32 66 66 33 37 66 31 37 62 39 34 62 39 32 62 31 63 34 65 31 38 30 31 61 32 35 33 36 32 37 32</StringToSignBytes><CanonicalRequest>PUT
/test-bucket.com/terraform-boot.log

date:Thu, 03 Aug 2017 08:42:56 UTC
host:s3-eu-west-1.amazonaws.com
x-amz-acl:public-read
x-amz-content-sha256:61c54136ff0a88b36398dfc4f450cd31d9d8a8b8680195e743b7f0d11f5f1a89
x-amz-date:20170803T084256Z

date;host;x-amz-acl;x-amz-content-sha256;x-amz-date
61c54136ff0a88b36398dfc4f450cd31d9d8a8b8680195e743b7f0d11f5f1a89</CanonicalRequest><CanonicalRequestBytes>50 55 54 0a 2f 74 66 73 74 61 74 65 2d 62 75 63 6b 65 74 2e 63 6f 6d 2f 74 65 72 72 61 66 6f 72 6d 2d 62 6f 6f 74 2e 6c 6f 67 0a 0a 64 61 74 65 3a 54 68 75 2c 20 30 33 20 41 75 67 20 32 30 31 37 20 30 38 3a 34 32 3a 35 36 20 55 54 43 0a 68 6f 73 74 3a 73 33 2d 65 75 2d 77 65 73 74 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 61 63 6c 3a 70 75 62 6c 69 63 2d 72 65 61 64 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 36 31 63 35 34 31 33 36 66 66 30 61 38 38 62 33 36 33 39 38 64 66 63 34 66 34 35 30 63 64 33 31 64 39 64 38 61 38 62 38 36 38 30 31 39 35 65 37 34 33 62 37 66 30 64 31 31 66 35 66 31 61 38 39 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 31 37 30 38 30 33 54 30 38 34 32 35 36 5a 0a 0a 64 61 74 65 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 61 63 6c 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 36 31 63 35 34 31 33 36 66 66 30 61 38 38 62 33 36 33 39 38 64 66 63 34 66 34 35 30 63 64 33 31 64 39 64 38 61 38 62 38 36 38 30 31 39 35 65 37 34 33 62 37 66 30 64 31 31 66 35 66 31 61 38 39</CanonicalRequestBytes><RequestId>B6FC666B6A05A5B4</RequestId><HostId>z/gQAjiXmFZccY+/a1L1hvsEVBn2/GycyGKRdVZN/1oV+0KtyM2IamAuzp60B9Nfs8m8RaxvlQw=</HostId></Error>

Umm, silly question but... Why don't you just use the AWS Command-Line Interface (CLI)? — John Rotenstein
– John Rotenstein, Commented Aug 3, 2017 at 10:40
I have a similar post for GET. Maybe you can use that one as a base stackoverflow.com/questions/40536928/… — Titi Wangsa bin Damhore
– Titi Wangsa bin Damhore, Commented Aug 3, 2017 at 11:39

Germain DENIS · Accepted Answer · 2017-08-04 08:41:12Z

1

Finnally i'm using a pyhton script much more easier this one for those who want to know :

#!/usr/bin/env python

import os
import boto
import sys
import boto.s3.connection
from boto.s3.key import Key

try:

conn = boto.s3.connect_to_region('eu-west-1',
aws_access_key_id = 'XXXX',
aws_secret_access_key = 'XXXX',
# host = 's3-website-us-east-1.amazonaws.com',
# is_secure=True,               # uncomment if you are not using ssl
calling_format = boto.s3.connection.OrdinaryCallingFormat(),
)

bucket = conn.get_bucket('test-bucket.com')
key_name = (sys.argv[1]) #argv[1] = the path of the file i want to upload
print (sys.argv[1])
path = '' #Directory Under which file should get upload
full_key_name = os.path.join(path, key_name)
k = bucket.new_key(full_key_name)
k.set_contents_from_filename(key_name)

except Exception,e:
print str(e)
print "error"

edited Aug 4, 2017 at 8:41

answered Aug 3, 2017 at 13:18

Germain DENIS

611 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

upload file on S3 using shell script

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related