Upload data to AWS S3 using linux shell script

Question

I have written shell script to upload data to s3 using CURL request but I am getting error I have tried so many ways but nothing work out for me. Used below CURL request

Upload the file to S3

curl -X PUT
-T "${FILE_PATH}"
-H "Host: ${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com"
-H "Content-Type: application/octet-stream"
-H "X-Amz-Content-SHA256: UNSIGNED-PAYLOAD"
-H "X-Amz-Date: ${REQUEST_DATE}"
-H "Authorization: AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY}/${REQUEST_SHORT_DATE}/${S3_REGION}/s3/aws4_request,SignedHeaders=host;content-type;x-amz-content-sha256;x-amz-date,Signature=${SIGNATURE}"
"https://${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com"

#!/bin/bash

# AWS Credentials
AWS_ACCESS_KEY="xxxxxxxxxxxx"
AWS_SECRET_KEY="yyyyyyyyyyyyyyyy"

# S3 Bucket details
S3_BUCKET="xxxxxx"
S3_REGION="ap-south-1"
FILE_PATH="/home/user/data.sh"

# Set the date and time for the request
REQUEST_DATE=$(date -u "+%Y%m%dT%H%M%SZ")
REQUEST_SHORT_DATE=$(echo $REQUEST_DATE | cut -c 1-8)

# Generate the signature
SIGNING_KEY=$(printf "${REQUEST_SHORT_DATE}\n${S3_REGION}\ns3\naws4_request\n${AWS_SECRET_KEY}" | \
              openssl sha256 -hmac "${REQUEST_SHORT_DATE}" -binary | \
              xxd -p -c 256)
CANONICAL_REQUEST=$(printf "PUT\n/${S3_OBJECT_KEY}\n\nhost:${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com\n\nhost\ncontent-type:application/octet-stream\n\ndate:${REQUEST_DATE}\n\nx-amz-content-sha256:UNSIGNED-PAYLOAD\nx-amz-date:${REQUEST_DATE}\n\n/${S3_OBJECT_KEY}")
CANONICAL_REQUEST_HASH=$(printf "${CANONICAL_REQUEST}" | openssl sha256 | awk '{print $2}')
STRING_TO_SIGN=$(printf "AWS4-HMAC-SHA256\n${REQUEST_DATE}\n${REQUEST_SHORT_DATE}/${S3_REGION}/s3/aws4_request\n${CANONICAL_REQUEST_HASH}")
SIGNATURE=$(printf "${STRING_TO_SIGN}" | openssl sha256 -mac HMAC -macopt "hexkey:${SIGNING_KEY}" -binary | xxd -p -c 256)

# Upload the file to S3
curl -X PUT \
  -T "${FILE_PATH}" \
  -H "Host: ${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com" \
  -H "Content-Type: application/octet-stream" \
  -H "X-Amz-Content-SHA256: UNSIGNED-PAYLOAD" \
  -H "X-Amz-Date: ${REQUEST_DATE}" \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=${AWS_ACCESS_KEY}/${REQUEST_SHORT_DATE}/${S3_REGION}/s3/aws4_request,SignedHeaders=host;content-type;x-amz-content-sha256;x-amz-date,Signature=${SIGNATURE}" \
  "https://${S3_BUCKET}.s3.${S3_REGION}.amazonaws.com"`

####error####

SignatureDoesNotMatchThe request signature we calculated does not match the signature you provided. Check your key and signing method.AKIAZP5IDSQC7FNOOQCFAWS4-HMAC-SHA256 20230301T060846Z 20230301/ap-south-1/s3/aws4_request 3ae12e94bc2d6de162fc7a12aa4d4baf25974b6f3920a7433cc0d5f834bac4d23d8a12be533618007718a22e41fce6dd10d565834553fbf1b9c79ac2cd920d6d41 57 53 34 2d 48 4d 41 43 2d 53 48 41 32 35 36 0a 32 30 32 33 30 33 30 31 54 30 36 30 38 34 36 5a 0a 32 30 32 33 30 33 30 31 2f 61 70 2d 73 6f 75 74 68 2d 31 2f 73 33 2f 61 77 73 34 5f 72 65 71 75 65 73 74 0a 33 61 65 31 32 65 39 34 62 63 32 64 36 64 65 31 36 32 66 63 37 61 31 32 61 61 34 64 34 62 61 66 32 35 39 37 34 62 36 66 33 39 32 30 61 37 34 33 33 63 63 30 64 35 66 38 33 34 62 61 63 34 64 32PUT /data.sh

content-type:application/octet-stream host:vedic.s3.ap-south-1.amazonaws.com x-amz-content-sha256:UNSIGNED-PAYLOAD x-amz-date:20230301T060846Z

content-type;host;x-amz-content-sha256;x-amz-date UNSIGNED-PAYLOAD50 55 54 0a 2f 64 61 74 61 2e 73 68 0a 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3a 61 70 70 6c 69 63 61 74 69 6f 6e 2f 6f 63 74 65 74 2d 73 74 72 65 61 6d 0a 68 6f 73 74 3a 76 65 64 69 63 2e 73 33 2e 61 70 2d 73 6f 75 74 68 2d 31 2e 61 6d 61 7a 6f 6e 61 77 73 2e 63 6f 6d 0a 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 44 0a 78 2d 61 6d 7a 2d 64 61 74 65 3a 32 30 32 33 30 33 30 31 54 30 36 30 38 34 36 5a 0a 0a 63 6f 6e 74 65 6e 74 2d 74 79 70 65 3b 68 6f 73 74 3b 78 2d 61 6d 7a 2d 63 6f 6e 74 65 6e 74 2d 73 68 61 32 35 36 3b 78 2d 61 6d 7a 2d 64 61 74 65 0a 55 4e 53 49 47 4e 45 44 2d 50 41 59 4c 4f 41 442FVFF45HZTKJR1W5axa+Me22CvWnXJ4j/9rAcKNjndTQGeWJIZGo3F+FpATI+R9olEsnLLxut4RzT5kZeJVJX90WvrQ=

Can you use the AWS CLI tool instead? AWS solves for you the signing feature, so you don't need to worry, and your script will have fewer lines ;-) docs.aws.amazon.com/cli/latest/userguide/… docs.aws.amazon.com/cli/latest/reference/s3/cp.html — michail_w
– michail_w, Commented Mar 1, 2023 at 8:56

Nick Allen · Accepted Answer · 2023-11-14 19:41:36Z

This bash script worked for me: https://gist.github.com/vszakats/2917d28a951844ab80b1

Here are the current (2023-11-14) contents of the link in case it disappears someday.

    #!/bin/sh
    
    # To the extent possible under law, Viktor Szakats
    # has waived all copyright and related or neighboring rights to this
    # script.
    # CC0 - https://creativecommons.org/publicdomain/zero/1.0/
    # SPDX-License-Identifier: CC0-1.0
    
    # THIS SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    # IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    # FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT. IN NO EVENT SHALL
    # THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
    # IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
    # CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    
    # Upload a file to Amazon AWS S3 (and compatible) using Signature Version 4
    #
    # docs:
    #   https://docs.aws.amazon.com/general/latest/gr/sigv4-create-canonical-request.html
    #   https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
    #
    # requires:
    #   curl, openssl 1.x or newer, GNU sed, LF EOLs in this file
    
    # shellcheck disable=SC2317
    
    # shellcheck disable=SC3040
    set -o errexit -o nounset; [ -n "${BASH:-}${ZSH_NAME:-}" ] && set -o pipefail
    
    fileLocal="${1:-example-local-file.ext}"
    bucket="${2:-example-bucket}"  # AWS S3 bucket or full URL (with ending slash): https://localhost[:9000]/[bucket/]
    region="${3:-}"
    storageClass="${4:-STANDARD}"  # or 'REDUCED_REDUNDANCY'
    SSE="${5:-AES256}"; [ "${SSE}" = 'none' ] && SSE=''  # Server-side encryption: 'AES256' (default) or 'none'
    
    my_openssl() {
      if [ -f /usr/local/opt/[email protected]/bin/openssl ]; then
        /usr/local/opt/[email protected]/bin/openssl "$@"
      elif [ -f /usr/local/opt/openssl/bin/openssl ]; then
        /usr/local/opt/openssl/bin/openssl "$@"
      else
        openssl "$@"
      fi
    }
    
    my_sed() {
      if command -v gsed > /dev/null 2>&1; then
        gsed "$@"
      else
        sed "$@"
      fi
    }
    
    awsStringSign4() {
      kSecret="AWS4$1"
      kDate=$(printf         '%s' "$2" | my_openssl dgst -sha256 -hex -mac HMAC -macopt "key:${kSecret}"     2>/dev/null | my_sed 's/^.* //')
      kRegion=$(printf       '%s' "$3" | my_openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kDate}"    2>/dev/null | my_sed 's/^.* //')
      kService=$(printf      '%s' "$4" | my_openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kRegion}"  2>/dev/null | my_sed 's/^.* //')
      kSigning=$(printf 'aws4_request' | my_openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kService}" 2>/dev/null | my_sed 's/^.* //')
      signedString=$(printf  '%s' "$5" | my_openssl dgst -sha256 -hex -mac HMAC -macopt "hexkey:${kSigning}" 2>/dev/null | my_sed 's/^.* //')
      printf '%s' "${signedString}"
    }
    
    iniGet() {
      # based on: https://stackoverflow.com/questions/22550265/read-certain-key-from-certain-section-of-ini-file-sed-awk#comment34321563_22550640
      printf '%s' "$(my_sed \
        -n -E "/\[$2\]/,/\[.*\]/{/$3/s/(.*)=[ \\t]*(.*)/\2/p}" "$1")"
    }
    
    # Initialize access keys
    
    if [ -z "${AWS_CONFIG_FILE:-}" ]; then
      if [ -z "${AWS_ACCESS_KEY_ID:-}" ]; then
        >&2 echo '! AWS_CONFIG_FILE or AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY envvars not set.'
        exit 1
      else
        awsAccess="${AWS_ACCESS_KEY_ID}"
        awsSecret="${AWS_SECRET_ACCESS_KEY}"
        awsRegion="${AWS_DEFAULT_REGION:-eu-west-1}"
      fi
    else
      awsProfile='default'
    
      # Read standard aws-cli configuration file
      # pointed to by the envvar AWS_CONFIG_FILE
      awsAccess="$(iniGet "${AWS_CONFIG_FILE}" "${awsProfile}" 'aws_access_key_id')"
      awsSecret="$(iniGet "${AWS_CONFIG_FILE}" "${awsProfile}" 'aws_secret_access_key')"
      awsRegion="$(iniGet "${AWS_CONFIG_FILE}" "${awsProfile}" 'region')"
    fi
    
    # Initialize defaults
    
    if [ -z "${region}" ]; then
      region="${awsRegion}"
    fi
    
    >&2 echo "! Uploading..." "${fileLocal}" "->" "${bucket}" "${region}" "${storageClass}"
    >&2 echo "! | $(uname) | $(my_openssl version) | $(my_sed --version | head -1) |"
    
    # Initialize helper variables
    
    httpReq='PUT'
    authType='AWS4-HMAC-SHA256'
    service='s3'
    if [ "${bucket#https://*}" != "${bucket}" ] || \
       [ "${bucket#http://*}" != "${bucket}" ]; then
      fullUrl="${bucket}"
    else
      fullUrl="https://${bucket}.${service}.${region}.amazonaws.com/"
    fi
    pathRemote="$(printf '%s' "${fullUrl}" | sed -E 's|^https?://||g' | grep -o -E '/.*$' | cut -c 2-)"
    hostport="$(printf '%s' "${fullUrl}" | sed -E -e 's|^https?://||g' -e 's|/.*$||')"
    dateValueS=$(date -u +'%Y%m%d')
    dateValueL=$(date -u +'%Y%m%dT%H%M%SZ')
    if command -v file >/dev/null 2>&1; then
      contentType="$(file --brief --mime-type "${fileLocal}")"
    else
      contentType='application/octet-stream'
    fi
    
    # Try to URL-encode the filename we pass
    
    # based on: https://gist.github.com/jaytaylor/5a90c49e0976aadfe0726a847ce58736?permalink_comment_id=4043195#gistcomment-4043195
    # Very curl version dependent.
    fileRemote="$({ curl --silent --get / --data-urlencode "=${fileLocal}" --write-out '%{url}' 2>/dev/null || true; } | cut -c 3- | sed 's/+/%20/g')"
    if [ -z "${fileRemote}" ] || [ "${fileRemote}" = '/' ]; then
      # Needs trurl
      fileRemote="$({ trurl --accept-space "file:///${fileLocal}" 2>/dev/null || true; } | cut -c 9-)"
      if [ -z "${fileRemote}" ]; then
        # Needs python3
        fileRemote="$({ printf '%s' "${fileLocal}" | python3 \
          -c 'import sys; import urllib.parse as ul; sys.stdout.write(ul.quote_plus(sys.stdin.read()))' 2>/dev/null || true; } | sed 's/+/%20/g')"
        if [ -z "${fileRemote}" ]; then
          # Last resort, that will probably not work as expected, but better than an empty string
          fileRemote="${fileLocal}"
        fi
      fi
    fi
    
    # 0. Hash the file to be uploaded
    
    if [ -f "${fileLocal}" ]; then
      payloadHash=$(my_openssl dgst -sha256 -hex /dev/null | my_sed 's/^.* //')
    else
      >&2 echo "! File not found: '${fileLocal}'"
      exit 1
    fi
    
    # 1. Create canonical request
    
    # NOTE: order significant in ${headerList} and ${canonicalRequest}
    
    if [ -n "${SSE}" ]; then
      headerList='content-type;host;x-amz-content-sha256;x-amz-date;x-amz-server-side-encryption;x-amz-storage-class'
      ssehdr="\
    x-amz-server-side-encryption:${SSE}
    "
    else
      headerList='content-type;host;x-amz-content-sha256;x-amz-date;x-amz-storage-class'
      ssehdr=''
    fi
    
    canonicalRequest="\
    ${httpReq}
    /${pathRemote}${fileRemote}
    
    content-type:${contentType}
    host:${hostport}
    x-amz-content-sha256:${payloadHash}
    x-amz-date:${dateValueL}
    ${ssehdr}x-amz-storage-class:${storageClass}
    
    ${headerList}
    ${payloadHash}"
    
    # Hash it
    
    canonicalRequestHash=$(printf '%s' "${canonicalRequest}" | my_openssl dgst -sha256 -hex 2>/dev/null | my_sed 's/^.* //')
    
    # 2. Create string to sign
    
    stringToSign="\
    ${authType}
    ${dateValueL}
    ${dateValueS}/${region}/${service}/aws4_request
    ${canonicalRequestHash}"
    
    # 3. Sign the string
    
    signature=$(awsStringSign4 "${awsSecret}" "${dateValueS}" "${region}" "${service}" "${stringToSign}")
    
    # Upload
    
    curl --silent --location --proto-redir =https --request "${httpReq}" --upload-file "${fileLocal}" \
      --header "Content-Type: ${contentType}" \
      --header "Host: ${hostport}" \
      --header "X-Amz-Content-SHA256: ${payloadHash}" \
      --header "X-Amz-Date: ${dateValueL}" \
      --header "X-Amz-Server-Side-Encryption: ${SSE}" \
      --header "X-Amz-Storage-Class: ${storageClass}" \
      --header "Authorization: ${authType} Credential=${awsAccess}/${dateValueS}/${region}/${service}/aws4_request, SignedHeaders=${headerList}, Signature=${signature}" \
      "${fullUrl}${fileRemote}"
    
    return
    
    # Examples
    
    cat > 'test.xml' <<EOF
    <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    <test>
    </test>
    EOF
    
    export AWS_ACCESS_KEY_ID='<example-id>'
    export AWS_SECRET_ACCESS_KEY='<example-key>'
    
    ./s3-upload-aws4.sh 'test.xml' 'http://localhost:9000/example-bucket/' 'eu-west-1' '' 'none'
    ./s3-upload-aws4.sh 'test.xml' 'http://localhost:9000/example-bucket/' 'eu-west-1' ''
    ./s3-upload-aws4.sh 'test.xml' 'example-bucket' 'eu-west-1' ''

(credits to Viktor Szakats)

Collectives™ on Stack Overflow

Upload data to AWS S3 using linux shell script

Upload the file to S3

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Upload the file to S3

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related