I'd like to replace a variable in a script template by a public and private certificate.
For example, I've generated a harbor.crt public certificate and a harbor.key private key with the following command:
sudo openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout /data/harbor.key -out /data/harbor.crt -subj "/CN=$LOCAL_IP" -addext "subjectAltName=IP:127.0.0.1,IP:$LOCAL_IP"
In a template script, I've the following variables I'd like to replace with the above files:
CFG_HARBOR_CRT="CRT" # Harbor registry certificate
CFG_HARBOR_KEY="KEY" # Harbor registry key
To replace those values, I've tried to do something like that:
HARBOR_CRT=`sudo cat /data/harbor.crt`
HARBOR_KEY=`sudo cat /data/harbor.key`
sudo sed -i "s/CFG_HARBOR_CRT\=\"[^\"]*\"/CFG_HARBOR_CRT\=\"$HARBOR_CRT\"/g" ./template-script.sh
sudo sed -i "s/CFG_HARBOR_KEY\=\"[^\"]*\"/CFG_HARBOR_KEY\=\"$HARBOR_KEY\"/g" ./template-script.sh
But both commands failed on: sed: -e expression #1, char 70: unterminated s' command`
Is there a way to use sed command with unescaped variables ?
I suspect there's info missing here. Why use sed at all?
For the simple case, just replace the markers with file reads.
CFG_HARBOR_CRT="$(</data/harbor.crt)"
CFG_HARBOR_KEY="$(</data/harbor.key)"
That might mean you need to run the whole script with elevated priv's though, so I understand why you might not want to do that.
...do you need root to read those files?
If so, and if you don't want the whole script run as root, maybe this:
$: sed 's,^CFG_HARBOR_CRT="CRT",CFG_HARBOR_CRT="$(sudo cat /data/harbor.crt)",
s,^CFG_HARBOR_KEY="KEY",CFG_HARBOR_KEY="$(sudo cat /data/harbor.key)",' tmpf
CFG_HARBOR_CRT="$(sudo cat /data/harbor.crt)" # Harbor registry certificate
CFG_HARBOR_KEY="$(sudo cat /data/harbor.key)" # Harbor registry key
Switching / to , as the demarcation reduces leaning toothpick syndrome.
Switching `...` to $(...) improves flexibility, stability, readability, etc.
Pulling out of comments to get better visibility ...
Consider running the files through base64 and embedding the result into the script, then on the other end run base64 -d to decrypt the data and store in the target files.
Using base64 encoded data should eliminate most (all?) of the sed headaches of dealing with special characters and/or trying to find a sed script delimiter that's not in the data.
OP/Manitoba's reply comment:
That did the trick. I used HARBOR_CRT=$(sudo cat /data/harbor.crt | base64 -w 0) to convert certificate to B64 and echo $CFG_HARBOR_CRT | base64 --decode to decode.
Related
sed -i "s|{{PLACEHOLDER}}|${KEY_B64}|g" <path>
The command above is executed in a Gitlab CI runner and throws the following:
sed: unmatched '|'
I have double-checked the KEY_B64 environment variable, it is set and looks valid.
This variable is a base-64 encoded JWT token (Kubernetes secrets expect to be base-64 encoded.
What is really strange though is that this command works fine if I run it locally (Ubuntu 22.04) and replace the env variable with the output from echo -n <JWT_TOKEN> | base64.
Based on the error message, it seems that the env value might contain the delimiter, but changing it to anything else doesn't solve the problem. On top of that, the encoded value for sure doesn't include such symbols.
What could be the cause of the issue?
Updates:
Running sed --version outputs:
$ sed --version
This is not GNU sed version 4.0
Looking with the set -x option on, I can see that the encoded string includes newlines (outputted the variable in the pipeline logs).
Using printf %s $VAR did not solve the issue
Surprisingly, base64 doesn't support -w0
Using set -x revealed that for some reason the base64 command added line breaks to the output.
As base64 -w0 is not supported, I had to use the command below to remove the newlines in the base64 output, which solved the problem.
export MY_KEY_B64=$(echo -n $MY_KEY | base64 | tr -d \\n)
Note: as I was fairly told in the comments, using printf is preferable but in this case, it was not the cause of the issue so I did not modify this command to emphasize what cause the problem.
I'm using a macOS Sierra and want to sha256 each line in a file.
File:
test
example
sample
Intended output:
9F86D081884C7D659A2FEAA0C55AD015A3BF4F1B2B0B822CD15D6C15B0F00A08
50D858E0985ECC7F60418AAF0CC5AB587F42C2570A884095A9E8CCACD0F6545C
AF2BDBE1AA9B6EC1E2ADE1D694F41FC71A831D0268E9891562113D8A62ADD1BF
I've found a few ways to do this on Ubuntu, but Mac doesn't seem to have sha256 installed on it. From what is sounds like, you have to use something similar to shasum 256 but nothing seems to be working.
openssl dgst is available out-of-the-box on MacOS (and pretty much everywhere else as well), and can be easily combined with BashFAQ #1 practices for iterating line-by-line:
### INEFFICIENT SOLUTION
hashlines() {
while IFS= read -r line; do
printf '%s' "$line" \
| openssl dgst -sha256 \
| tr '[[:lower:]]' '[[:upper:]]' \
| sed -Ee 's/.*=[[:space:]]//'
done
}
That said, this is quite wildly inefficient when run on large files. If you need something that performs well, I'd write this in Python instead. Wrapped in the same shell function, with the same calling convention, that might look like:
### FAST SOLUTION
hashlines() {
python -c '
import sys, hashlib
for line in sys.stdin:
print(hashlib.sha256(line.rstrip("\n")).hexdigest().upper())'
}
In either case, usage is just hashlines < filename or hashlines <<< $'line one\nline two'.
I keep getting this error while trying to autoload a file to S3:
The request signature we calculated does not match the signature you provided. Check your key and signing method.
HMAC-SHA256s(){
KEY="$1"
DATA="$2"
shift 2
printf "$DATA" | openssl dgst -binary -sha256 -hmac "$KEY" | od -An -vtx1 | sed 's/[ \n]//g' | sed 'N;s/\n//'
}
HMAC-SHA256h(){
KEY="$1"
DATA="$2"
shift 2
printf "$DATA" | openssl dgst -binary -sha256 -mac HMAC -macopt "hexkey:$KEY" | od -An -vtx1 | sed 's/[ \n]//g' | sed 'N;s/\n//'
}
FILE_TO_UPLOAD=/var/www/cool/main.txt
BUCKET="temporaltestingstorage"
STARTS_WITH="Schiller/Zauberlehrling"
REQUEST_TIME=$(date +"%Y%m%dT%H%M%SZ")
REQUEST_REGION="eu-central-1"
REQUEST_SERVICE="s3"
REQUEST_DATE=$(printf "${REQUEST_TIME}" | cut -c 1-8)
AWS4SECRET="AWS4"$AWS_SECRET_KEY
ALGORITHM="AWS4-HMAC-SHA256"
EXPIRE="2015-01-01T00:00:00.000Z"
ACL="private"
POST_POLICY='{"expiration":"'$EXPIRE'","conditions": [{"bucket":"'$BUCKET'" },{"acl":"'$ACL'" },["starts-with", "$key", "'$STARTS_WITH'"],["eq", "$Content-Type", "application/octet-stream"],{"x-amz-credential":"'$AWS_ACCESS_KEY'/'$REQUEST_DATE'/'$REQUEST_REGION'/'$REQUEST_SERVICE'/aws4_request"},{"x-amz-algorithm":"'$ALGORITHM'"},{"x-amz-date":"'$REQUEST_TIME'"}]}'
UPLOAD_REQUEST=$(printf "$POST_POLICY" | openssl base64 )
UPLOAD_REQUEST=$(echo -en $UPLOAD_REQUEST | sed "s/ //g")
SIGNATURE=$(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256h $(HMAC-SHA256s $AWS4SECRET $REQUEST_DATE ) $REQUEST_REGION) $REQUEST_SERVICE) "aws4_request") $UPLOAD_REQUEST)
curl \
--limit-rate 300k \
--connect-timeout 120 \
-F "key=$STARTS_WITH" \
-F "acl=$ACL" \
-F "Content-Type=application/octet-stream" \
-F "x-amz-algorithm=$ALGORITHM" \
-F "x-amz-credential=$AWS_ACCESS_KEY/$REQUEST_DATE/$REQUEST_REGION/$REQUEST_SERVICE/aws4_request" \
-F "x-amz-date=$REQUEST_TIME" \
-F "Policy=$UPLOAD_REQUEST" \
-F "X-Amz-Signature=$SIGNATURE" \
-F "file=#"$FILE_TO_UPLOAD http://$BUCKET.s3.amazonaws.com/
am I missing something?
Thanks
Several things I suggest your to change in your script before it can be debugged properly:
"Double quote" every literal that contains spaces/metacharacters and every expansion: "$var", "$(command "$var")", "${array[#]}", "a & b". See
Quotes
Arguments
Words
Regarding the variable POST_POLICY : Single quotes ( i.e. ') cause everything between them to be taken literally by bash. In your script, expressions like "$EXPIRE", "$BUCKET", "$ACL" don't expand in single quotes. If you want to embed a ' inside a '...', write it as the four characters, '\'' : printf '%s\n' 'It'\''s a blast!'
Check if this problem exist in more places in your script, and modify accordingly.
By convention, environment variables (PATH, EDITOR, SHELL, ...) and internal shell variables (BASH_VERSION, RANDOM, ...) are fully capitalized. All other variable names should be lowercase. Since
variable names are case-sensitive, this convention avoids accidentally overriding environmental and internal variables.
echo has many portability problems, and should never be used with option flags. Use printf instead: printf 'name: %s\n' "$name".
See
http://wiki.bash-hackers.org/commands/builtin/echo
http://cfajohnson.com/shell/cus-faq.html#Q0b
http://www.in-ulm.de/~mascheck/various/echo+printf
Not so important, but worth mentioning: Don't use variables in printf's format string ( e.g. printf '%s' "$DATA" instead of printf "$DATA" ).
I would debug this by placing a set -x at the top of the file so that you can see all the command output.
I've suffered a lot lately from using quotation marks too much or too little in my bash scripts lately. I'm not an expert, but it seems to me you might be double wrapping some variables in quotations. That could create an incorrect digest.
Also, the way you're creating the digest seems a bit error prone since you're doing some modifications on the openssl output. There must be a way to avoid that our get openssl to create the digest as Amazon specifies. Going to peek at that.
Also, in the documentation they provide a way for you to test that you are creating the signature correctly. I would try out your signature logic on that and that would narrow it down to being a problem with the way you're doing the SHA256.
And here's a gist with some like minded individuals. Saw some blog posts that were outdated, and this gist seems to diverge from the documentation, but I appreciated their simple take on SHAing. Perhaps you should share your question there and ask them to update the gist.
I'm sorry this isn't more of an answer, but was too long for a comment. I hope it does get answered better! Always interested in learning more about openssl.
I am trying to get a base64 encoded sha1 hash in a windows batch file.
The first thing I tried was with perl:
perl -M"Digest::SHA1 qw(sha1_base64)" -e "open(F,shift) or die; binmode F; print sha1_base64(<F>), qq(=\n)" "test.mxf"
This works great, but only for small files. With big files it says "Out of memory".
Then I downloaded an openssl version for windows and tried this:
"C:\openssl.exe" dgst -sha1 -binary -out "hash_sha1.txt" "C:\test.mxf"
set /p hash_sha1=<"hash_sha1.txt"
del "hash_sha1.txt"
echo !hash_sha1!
echo -n '!hash_sha1!' | "C:\openssl.exe" enc -base64
But the output of the openssl method is different from the Perl output and I know that the Perl method produces the correct output. What do I have to change?
There's no -n parameter of echo so -n AND single quotes are part of the output.
The intermediate files and variables aren't needed, use piping.
The entire code:
openssl dgst -sha1 -binary "C:\test.mxf" | openssl enc -base64
If you create a Digest::SHA1 object, you can use the add method to calculate the hash incrementally
There is also no need to explicitly open files passed as command-line parameters. They are opened automatically using the built-in file handle ARGV, and can be read with the empoty diamond operator <>
perl -Mopen=IN,:raw -MDigest::SHA1 -e"$d=Digest::SHA1->new; $d->add($_) while <>; print $d->b64digest, qq{=\n}" 5GB.bin
This command line was quite happy to generate the SHA1 hash of a 5GB file, but if you are unlucky enough to have a very big file that contains no linefeeds then you will have to set a read block size with something like
local $/ = \(1024*1024)
I am attempting to use the Azure blob storage service from a bash script using the REST API. I know it is possible to accomplish this using various other tools or languages, however I'd like to do it as a bash script.
The script below is an attempt to list the blobs in an Azure storage container.
This script results in an authentication error. The signing string and headers look correct based on the REST API (reference) documentation. I suspect the problem may be in juggling the various parts of the signing process.
Has anyone successfully used bash and curl to access cloud storage resources like Azure or other providers?
#!/bin/bash
# List the blobs in an Azure storage container.
echo "usage: ${0##*/} <storage-account-name> <container-name> <access-key>"
storage_account="$1"
container_name="$2"
access_key="$3"
blob_store_url="blob.core.windows.net"
authorization="SharedKey"
request_method="GET"
request_date=$(TZ=GMT date "+%a, %d %h %Y %H:%M:%S %Z")
storage_service_version="2011-08-18"
# HTTP Request headers
x_ms_date_h="x-ms-date:$request_date"
x_ms_version_h="x-ms-version:$storage_service_version"
# Build the signature string
canonicalized_headers="${x_ms_date_h}\n${x_ms_version_h}"
canonicalized_resource="/${storage_account}/${container_name}"
string_to_sign="${request_method}\n\n\n\n\n\n\n\n\n\n\n\n${canonicalized_headers}\n${canonicalized_resource}\ncomp:list\nrestype:container"
# Decode the Base64 encoded access key, convert to Hex.
decoded_hex_key="$(echo -n $access_key | base64 -d -w0 | xxd -p -c256)"
# Create the HMAC signature for the Authorization header
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" | sed 's/^.*= //' | base64 -w0)
authorization_header="Authorization: $authorization $storage_account:$signature"
curl \
-H "$x_ms_date_h" \
-H "$x_ms_version_h" \
-H "$authorization_header" \
"https://${storage_account}.${blob_store_url}/${container_name}?restype=container&comp=list"
Update - The storage service error and the corresponding signing string that the script generated.
Following is what the storage service returns for the AuthenticationFailed error.
<?xml version="1.0" encoding="utf-8"?>
<Error>
<Code>AuthenticationFailed</Code>
<Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:27e6337e-52f3-4e85-98c7-2fabaacd9ebc
Time:2013-11-21T22:10:11.7029042Z</Message>
<AuthenticationErrorDetail>The MAC signature found in the HTTP request
'OGYxYjk1MTFkYmNkMCgzN2YzODQwNzcyNiIyYTQxZDg0OWFjNGJiZDlmNWY5YzM1ZWQzMWViMGFjYTAyZDY4NAo='
is not the same as any computed signature. Server used following string to sign:
'GET
x-ms-date:Thu, 21 Nov 2013 22:10:11 GMT
x-ms-version:2011-08-18
/storage_account_name/storage_container
comp:list
restype:container'
</AuthenticationErrorDetail>
</Error>
Next is the string_to_sign that the script generates.
GET\n\n\n\n\n\n\n\n\n\n\n\nx-ms-date:Thu, 21 Nov 2013 22:10:11 GMT\nx-ms-version:2011-08-18\n/storage_account_name/storage_container\ncomp:list\nrestype:container
I was able to get it working.
There were two things wrong with this code, the first, as Patrick Park noted, was replacing the echo -n with printf. The second was replacing the sed magic with the -binary option on openssl.
Compare the original:
signature=$(echo -n "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | sed 's/^.*= //' | base64 -w0)
with the fixed:
signature=$(printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt "hexkey:$decoded_hex_key" -binary | base64 -w0)
The echo change is needed because echo -n will not convert the \n into actual newlines.
The -binary change is needed because even though you are stripping off the bad part, openssl was still outputting the signature in ascii-encoded-hex, not in binary. So after it was passed to base64, the result was the b64 encoded version of the hex representation, instead of the raw value.
Use Fiddler (or an equivalent on your platform) to intercept the call to Windows Azure Storage. On failure, this will show you the string that the Storage Service used to authenticate the call and you can compare this with the one you used.
Looking at the REST API documentation and your code above, I believe there's an issue with the way you're constructing canonicalized_resource string. You're missing the query parameters in that string. Your canonicalized_resource string should be:
canonicalized_resource="/${storage_account}/${container_name}\ncomp:list\nrestype:container"
It looks like openssl dgst does not generate proper HMAC for you.
I wrote a simple program in C that does the following:
Takes base64-encoded key from the command line and decodes it into binary.
Reads string to sign from standard input.
Uses libcrypto HMAC() routine to generate the signature.
base64-encodes the signature and prints the result to standard output.
I then replaced openssl dgst pipeline in your script with the call to my program and it did the trick.
Please note that the output you are getting from Azure is XML-wrapped and base-64 encoded, so you'll need to come up with some sort of parsing/conversion code for it.
use printf instead of echo (it works for me)
for example:
SIGNATURE=printf "$string_to_sign" | openssl dgst -sha256 -mac HMAC -macopt hexkey:$HEXKEY -binary | base64 -w0