I would like to authenticate my S3 requests using this method. For this purpose I have adapted scripts presented here. Here is how it looks like:
#!/bin/sh
file="$2"
bucket="$1"
resource="/${bucket}/${file}"
contentType="$3"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="$AWS_ACCESS_KEY"
s3Secret="$AWS_SECRET_ACCESS_KEY"
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -v -H "Host: ${bucket}.s3.amazonaws.com" -H "Date: ${dateValue}" -H "Content-Type: ${contentType}" -H "Authorization: AWS ${s3Key}:${signature}" https://${bucket}.s3.amazonaws.com/${file}
My file permissions are like this:
and file content type is:
Here is how do I call my script:
./getfile1.sh tmp666 7b5879dd9b894f7fc5a14b895f01abd1.png image/png
Unfortunately I am getting SignatureDoesNotMatch error although StringToSign returned by AWS is matching mine:
<StringToSign>GET
image/png
Wed, 28 Sep 2016 12:46:14 +0200
/tmp666/7b5879dd9b894f7fc5a14b895f01abd1.png</StringToSign>
My IAM user has admin permissions. What am I doing wrong? I have spent half of day trying to solve it and feel powerless.
Related
I'm trying to upload the data to aws s3 using following script and getting below error.
Example Script
#!/usr/bin/sh
file_to_upload="/home/sraj/Hello10.txt"
bucket="mybucket"
filepath="/${bucket}/${file_to_upload}"
contentType='application\/x-compressed-tar'
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
signature_string="PUT\n\n${contentType}\n${dateValue}\n${filepath}"
s3_access_key=xxxxxxxxx
s3_secret_key=yyyyyyyyy
signature_hash=`echo -en ${signature_string} | openssl sha256 -hmac ${s3_secret_key} -binary | base64`
echo "${s3_access_key} : ${s3_secret_key} : ${signature_hash}"
curl -X PUT -T "${file_to_upload}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3_access_key}:${signature_hash}" \
https://${bucket}.s3.amazonaws.com/${file_to_upload}
I think this is related to I AM user KMS encryption option.
Kindly help me how to solve above issue. I'm not much idea about curl and s3 also.
It will very helpful for sample code.
Some of them mentioned this is related to Signature version4 issue but I don't have much idea about fix this one.
The error says Invalid Argument.
I found this link, I have no idea if it will help.
https://www.gyanblog.com/aws/how-upload-aws-s3-curl/#curl-the-savior`
The only difference I see is the dateValue.
I do not know if that matters, but just in case, I posted this.
# about the file
file_to_upload=<file you want to upload>
bucket=<your s3 bucket name>
filepath="/${bucket}/${file_to_upload}"
# metadata
contentType="application/x-compressed-tar"
dateValue=`date -R`
signature_string="PUT\n\n${contentType}\n${dateValue}\n${filepath}"
#s3 keys
s3_access_key=<your s3 access key>
s3_secret_key=<your s3 secret key>
#prepare signature hash to be sent in Authorization header
signature_hash=`echo -en ${signature_string} | openssl sha1 -hmac ${s3_secret_key} -binary | base64`
# actual curl command to do PUT operation on s3
curl -X PUT -T "${file_to_upload}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3_access_key}:${signature_hash}" \
https://${bucket}.s3.amazonaws.com/${file_to_upload}
I realise there are quite a few posts that show how to capture output of curl or grep, but haven't been able to find any post where I can capture the value of a group in the grep's regular expression.
I need to capture output of the following command in a variable inside a shell script.
I am working on API calls of Qlik tool. The way the APIs work, is that you first have to call the login operation, which returns a session key and then use this session key in the subsequent API calls, like this
https://help.qlik.com/en-US/enterprise-manager/November2021/Content/EnterpriseManager/EnterpriseManager_APIGuide/CurlAPI/api_login.htm
cURL example
Active Directory request
curl -i -k --header "Authorization: Basic cWFAcWE6cWE=" https://computer.network.net/attunityenterprisemanager/api/v1/login
Response
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
EnterpriseManager.APISessionID: J3cKzWIbi_w6Fr1G-tO03Q
Date: Mon, 26 Dec 2016 17:02:01 GMT
And then use the EnterpriseManager.APISessionID
https://help.qlik.com/en-US/enterprise-manager/November2021/Content/EnterpriseManager/EnterpriseManager_APIGuide/CurlAPI/api_aem_getserver.htm
So, I am trying to get this EnterpriseManager.APISessionID value in a variable, so I can use it in the script to call other operations.
If I run the below command on the command line it prints the value on the terminal,
curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login | grep '(EnterpriseManager.APISessionID: )(.*)' | echo "${BASH_REMATCH[2]}"
This is the output on command line
eu_3qrEWJFAz_4hL7bOIvA
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
but if I put it in a .sh file and execute it, then the variable value is empty
API_SESSION_ID=$(curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login | grep 'EnterpriseManager.APISessionID: ')
API_SESSION_ID2=$(curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login | grep '(EnterpriseManager.APISessionID: )(.*)')
#API_SESSION_ID=$(curl -i -k --header "Authorization: Basic mybase64idpwvalue" https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login) | grep '(EnterpriseManager.APISessionID: )(.*)' | echo "${BASH_REMATCH[2]}"
echo "API_SESSION_ID : ${API_SESSION_ID}"
echo "API_SESSION_ID2 : ${API_SESSION_ID2}"
I tried some other combinations of the command in the script, but am unable to capture the value in a variable. It is always empty
How can I get that populated?
Thanks
First, I don't know where eu_3qrEWJFAz_4hL7bOIvA comes from in your example. I'll assume you meant J3cKzWIbi_w6Fr1G-tO03Q.
Second, your command line output doesn't make sense, unless you played around with the shell and some global state remains. When I run it, I get a empty output, which is what I expect.
Third, BASH_REMATCH is a bash variable that captures the result of regex matching from bash operators ([[ string =~ regex ]]). It's not aware of grep, perl, sed, or any other programs using regex.
That's why when I run your command I get no output - BASH_REMATCH is empty.
Last, your grep expression uses unquoted parentheses, and you didn't pass -E, so it looks for parentheses in the text, and it wouldn't find them. So even if BATCH_REMATCH was aware of grep, grep would still return an empty result.
All that said, sed does support capture:
API_SESSION_ID="$( \
curl -i -k \
--header "Authorization: Basic mybase64idpwvalue" \
https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login \
| sed -n 's/\(EnterpriseManager.APISessionID: \)\(.*\)/\2/p' \
)"
echo "API_SESSION_ID : ${API_SESSION_ID}"
That \2 represents the second capture group. But you don't actually need the first:
API_SESSION_ID="$( \
curl -i -k \
--header "Authorization: Basic mybase64idpwvalue" \
https://qlik-qem-dev.abc.com/attunityenterprisemanager/api/v1/login \
| sed -n 's/EnterpriseManager.APISessionID: \(.*\)/\1/p' \
)"
echo "API_SESSION_ID : ${API_SESSION_ID}"
BASH_REMATCH isn't related to grep. It must be called after regex matching.
GNU grep (perl positive look behind):
echo "$example_header" |
grep -Po '(?<=^EnterpriseManager\.APISessionID: ).*'
Pure bash:
[[ $example_header =~ (^|$'\n')(EnterpriseManager\.APISessionID: )([^$'\n']*) ]]
echo "${BASH_REMATCH[3]}"
Both print:
J3cKzWIbi_w6Fr1G-tO03Q
For input:
example_header=\
'Response
HTTP/1.1 200 OK
Content-Length: 0
Content-Type: text/html
Server: Microsoft-HTTPAPI/2.0
EnterpriseManager.APISessionID: J3cKzWIbi_w6Fr1G-tO03Q
Date: Mon, 26 Dec 2016 17:02:01 GMT'
# Or: example_header=$(curl -i -k --header "Authorization: Basic cWFAcWE6cWE=" https://computer.network.net/attunityenterprisemanager/api/v1/login)
Don't forget to escape . and to match the start of the line (to avoid matching NotEnterpriseManager...).
I'm making a curl request to a REST API and now I want to save the HTTP answer code, just if an error occurs, to a logfile and the API answer to another file if there is no error occuring.
I'm trying it with:
error=$(curl -v -o "test.json" -H "Authorization: Basic ABCDEF" "https://api.abc.com")
and
error=$(curl --fail -o "test.json" -H "Authorization: Basic ABCDEF" "https://api.abc.com")
If I make an if [0 -eq $? ] after the curl request with --fail I can detect that an error occurred but I am not able to save the HTTP error to a log.
Thanks.
Since according man curl the option --fail
... is not fail-safe and there are occasions where non-successful response codes will slip through, especially when authentication is involved (response codes 401 and 407).
you could use --write-out and http_code.
The numerical response code that was found in the last retrieved HTTP(S) or FTP(s) transfer.
I.e.:
ERROR=$(curl --silent --fail --header "Authorization: Basic ABCDEF" "https://api.abc.com" --output "test.json" --write-out "%{http_code}")
Welcome to stackoverflow.
This should do the trick.
#send all output to file named out
curl -v -o "test.json" -H "Authorization: Basic ABCDEF" "https://api.abc.com" >out 2>&1
# find HTTP/2 code in the output
error=`grep "HTTP/2" out | tail -1 | rev | cut -c1-5 | rev`
# print the error code
echo $error
Example run:
mamuns-mac:jenkins xmrashid$ ./get_error.sh
503
mamuns-mac:jenkins xmrashid$
Good Luck.
I'm using a script dump-to-s3.sh to put database dumps in a S3 bucket.
When triggered manually it works perfectly but when I trigger it via this cron (as root crontab) it fails with the following error message:
crontab
31 12 * * * /home/dokku/.mongodb/dump-to-s3.sh
error from CURL
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
Check your key and signing method.</Message>...
dump-to-s3.sh
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
#cd to dump-folder
cd /dump/folder
file="mydump.tar.gz"
bucket="mybucket"
resource="/${bucket}/dumps/${file}"
contentType="application/x-compressed-tar"
dateValue=`date -R`
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key="xxxxxxxxxxxxxxxx"
s3Secret="xxxxxxxxxxxxxxxx"
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -X PUT -T "${file}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/dumps/${file}
I think you might need to run bash manually, e.g.
/bin/bash /home/dokku/.mongodb/dump-to-s3.sh
The -en options for echo don't work in the regular shell; it might just be a bash extension.
In sh:
$ sh
sh-3.2$ echo -en foo
-en foo
sh-3.2$
In bash:
$ bash
bash-3.2$ echo -en foo
foobash-3.2$
I have created an IAM user with access to only one bucket. I have tested the credentials and permissions through web and python boto. Its working fine.
Now I have requirement to use these credentials and download the private file from that bucket through curl.
signature="$(echo -n "GET" | openssl sha1 -hmac "f/rHQ8yCvPthxxxxxxxXxxxx" -binary | base64)"
date="$(LC_ALL=C date -u +"%a, %d %b %Y %X %z")"
curl -H "Host: my-bucket.s3.amazonaws.com" -H "Date: $date" -H "Authorization: AWS 'XXXAJX2NY3QXXX35XXX':$signature" -H "Content-Type: 'text/plain'" https://my-bucket.s3.amazonaws.com/path/to_file.txt
but i am getting the following error:
InvalidAccessKeyIdThe AWS Access Key Id you provided does not exist in our records.
Please help, how do I download the file using curl ? Is there anything am I missing or its not possible through curl command?
Thanks!
Following is the example on how you can download with s3 curl script,
#!/bin/sh
file=path/to/file
bucket=your-bucket
resource="/${bucket}/${file}"
contentType="application/x-compressed-tar"
dateValue="`date +'%a, %d %b %Y %H:%M:%S %z'`"
stringToSign="GET
${contentType}
${dateValue}
${resource}"
s3Key=xxxxxxxxxxxxxxxxxxxx
s3Secret=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
signature=`/bin/echo -en "$stringToSign" | openssl sha1 -hmac ${s3Secret} -binary | base64`
curl -H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
Hope it helps.