How to pass "escape hell" to docker run - bash

I'm trying to pass several commands to an alpine container from powershell\cmd. The problem is: commands have quotes\single quotes\escaped quotes.
Sample command, the actual command is one line, I'm splitting it so it is easier to look at:
docker run neilpang/acme.sh bin/sh -c --%
"acme.sh --issue --dns dns_azure --dnssleep 10 --force -d "domain";
openssl pkcs12 -inkey /acme.sh/domain/domain.key -in /acme.sh/domain/domain.cer -export -out pfx -password pass:password;
curl -X POST -d "{\"certName\":\"certName1\",\"certData\":\"$(tail -n +2 /acme.sh/domain/domain.cer > b64; head -n -1 b64)\"}" url -H Content-Type:application/json;
curl -X POST -d "{\"certName\":\"certName2\",\"certData\":\"$(openssl base64 -in pfx -out b64; cat b64)\"}" url -H Content-Type:application/json"
What I've tried:
Use single quotes around the whole expression (after --%) returns something like quoted string not closed
Using single quotes around json works fine, but $() doesnt get interpreted, so I dont get proper output
escaping " before\after json
without " before\after json, results in some garbage
I never got it to work without --%, but it appears ps does some string manipulations even when using --%
did some experiments with cmd, no luck either.
I'm open to any way of making this work :)
for those requesting errors, here's a sample:
-n: line 1: syntax error: unexpected end of file (expecting ")") # quotes around expression
-p: line 1: syntax error: unterminated quoted string # single quotes around expression
no error, but certData empty # no quotes around json
no quotes in json >> not valid json >> doesnt get parsed # escaped quotes around json
executing the strings i'm trying to pass inside the containers results in the working "solution", whereas trying to pass them through docker run results in different defects described above. I'm not sure where these arise, from powershell (although, i firmly believe --% does the work here, and powershell passes everything properly), docker or bash itself. the closest i got to working is the ' version, but it doesnt evaluate expression inside $(), so that the problem

You'll find the quoting issues significantly easier if you turn your very long, very involved command line into a shell script, package it into a custom image, and then run that.
Shell script, let's call it run_acme.sh:
#!/bin/sh
acme.sh --issue --dns dns_azure --dnssleep 10 --force -d "domain"
openssl pkcs12 \
-inkey /acme.sh/domain/domain.key \
-in /acme.sh/domain/domain.cer \
-export \
-out pfx \
-password pass:password
CER_B64=$(tail -n +2 /acme.sh/domain/domain.cer | head -n -1)
curl -X POST \
--data-binary "{\"certName\":\"certName1\",\"certData\":\"$CER_B64\"}" \
-H 'Content-Type: application/json'
url
PFX_B64=$(openssl base64 -in pfx)
curl -X POST \
--data-binary "{\"certName\":\"certName1\",\"certData\":\"$PFX_B64\"}" \
-H 'Content-Type: application/json'
url
Dockerfile:
FROM neilpang/acme.sh
COPY run_acme.sh /bin
CMD ["/bin/run_acme.sh"]
Then you can docker build this like any other custom image, and docker run it, and the very long command line will be baked into the shell script in your image.

I'd suggest using splatting to make this significantly cleaner to maintain. Note: I'm uncertain whether your shell script needs to escape the quotes around the -d parameter.
$curl1 = '"{\"certName\":\"certName1\",' +
'\"certData\":\"$(tail -n +2 /acme.sh/domain/domain.cer > b64; head -n -1 b64)\"}"'
$curl2 = '"{\"certName\":\"certName2\",' +
'\"certData\":\"$(openssl base64 -in pfx -out b64; cat b64)\"}"'
$dockerArgs = #(
'run', 'neilpang/acme.sh', 'bin/sh', '-c'
'"acme.sh --issue --dns dns_azure --dnssleep 10 --force -d "domain";' +
'openssl pkcs12 -inkey /acme.sh/domain/domain.key -in /acme.sh/domain/domain.cer ' +
'-export -out pfx -password pass:password;' +
"curl -X POST -d $curl1 url -H Content-Type:application/json;" +
"curl -X POST -d $curl2 url -H Content-Type:application/json"""
)
docker.exe #dockerArgs
The powershell escape character is a grave-accent ` or you could double-up on the quotes "" to escape double-quotes "`"".

Related

Passing key and pem file data in curl command not working

I am trying to invoke the API using the curl command but unfortunately, I won't be able to keep the files. I am trying to pass the .key and .pem file's data in the command but I am not able to pass that correctly. Below is my command in my .sh file:
response=$(curl --key "$5" --cert "$6" -k -X "$2" -d "$payload" "$4")
I am calling the script below way:
key="${key}"
pem="${pem}"
bash ./Integration1.sh Provision POST "$payload" https://some-api.com/pr "$key" "$pem"
It gives the below error:
curl: (58) could not load PEM client certificate from -----BEGIN CERTIFICATE-----
This command works fine if I pass the file directly so, is there any way to pass the data via string variables in the curl command?
If you only have your key data in a variable and can't write it to a file yourself for some reason, an alternative solution is to use process substitution.
bash ./Integration1.sh Provision POST "$payload" https://some-api.com/pr \
<(printf '%s' "$key") \
<(printf '%s' "$pem")
This requires bash and still uses files under the hood, but it doesn't require you to manage the files yourself or know where they're located.
--key and --cert take the name of a file containing certificate data, not the certificate data itself.
... "$(cat my_client.key)" "$(cat my_client.pem)"
Should just be
... my_client.key my_client.pem

CURL request fails with error msg "failed to decrypt request"

I have implemented a CURL script to execute API call which requires encrypted string as data, But it gets failed with 'failed to decrypt request'. I have tried with python also (requests and Crypto modules) but failed with same error message.
#!/usr/bin/env bash
output=$(openssl enc -aes-256-cbc -pass pass:xxxxxxx -in msg.txt -base64)
curl -X POST \
"https://api.swing.tradesmartonline.in/api/v1/login-token" \
-H "Content-Type: application/json" \
-d '{"data":"$output","app":"APPLICATION_ID"}'
Output is: {"status":"false","data":"","error_msg":"failed to decrypt
request."}
Where as I am able to decrypt the message in local using below command
echo "$output" | openssl enc -aes-256-cbc -base64 -pass pass:xxxxxxx -d
Thanks in advance
The shell will not expand your variables in a single quoted string. Either:
Use double quotes and escape embedded double quotes with \":
"{\"data\":\"$output\",\"app\":\"APPLICATION_ID\"}"
Use single quotes for static text and double quotes for what contain variables (this means less escaping):
'{"data":"'"$output"'","app":"APPLICATION_ID"}'

Passing Variables in CURL GET

I'm trying to write a sh script that reads data from a JSON, and by parsing, pass the data as variables for a curl request (GET)
The parsing is done correctly but the CURL request continues to give problems.
I parse the variables, after opening the JSON, with the help of JQ:
ruleId=$($JSON | jq '.[].ruleId')
alert=$($JSON | jq '.[].alertName')
...
I'm passing the variables in this way:
curl --data-urlencode "ruleId=$ruleId" --data-urlencode "newLevel=$newLevel" --data-urlencode "url=$url" --data-urlencode "urlIsRegex=$urlIsRegex" --data-urlencode "enabled=$enabled" --data-urlencode "parameter=$parameter" --data-urlencode "evidence=$evidence" "http://localhost:8090/JSON/alertFilter/action/addGlobalAlertFilter"
The error that i get back is:
curl: (3) URL using bad/illegal format or missing URL
Can you help me?
Here is my test script:
!/bin/sh
#set -e
# Info to test this script
# 1. Start docker
# 2. Run this command from terminal: docker run -u zap -p 8080:8090 -i owasp/zap2docker-stable zap.sh -daemon -host 0.0.0.0 -port 8080 -config api.disablekey=true -config api.addrs.addr.name=.* -config api.addrs.addr.regex=true
# 2. Get the JSON from api (test for now)
# 3. bash filter.sh
# 4. Check for filter to be set with browser http://localhost:8080/UI/alertFilter/ => view global filter
#open example JSON file (API) and check for alert filter list
curl -s 'https://api.npoint.io/c29e3a68be632f73fc22' > whitelist_tmp.json
whitelist=whitelist_tmp.json
#Parser WITOUT loop
ruleId=$($whitelist | jq '.[].ruleId')
alert=$($whitelist | jq '.[].alertName')
newLevel=$($whitelist | jq '.[].newLevel')
url=$($whitelist | jq '.[].url')
urlIsRegex=$($whitelist | jq '.[].urlIsRegex')
enabled=$($whitelist | jq '.[].enabled')
parameter=$($whitelist | jq '.[].parameter')
evidence=$($whitelist | jq '.[].evidence')
#Fist test-query WITHOUT url-encoding => not working
#curl -X -4 --retry 2 --retry-connrefused --retry-delay 3 "http://localhost:8080/JSON/alertFilter/action/addGlobalAlertFilter/?ruleId=${ruleId}\&newLevel=${newLevel}\&url=${url}\&urlIsRegex=${urlIsRegex}\&parameter=${parameter}\&enabled=${enabled}\&evidence=${evidence}"
#Second test-query WITH url-encoding
curl --data-urlencode "ruleId=$ruleId" --data-urlencode "newLevel=$newLevel" --data-urlencode "url=$url" --data-urlencode "urlIsRegex=$urlIsRegex" --data-urlencode "enabled=$enabled" --data-urlencode "parameter=$parameter" --data-urlencode "evidence=$evidence" "http://localhost:8080/JSON/alertFilter/action/addGlobalAlertFilter"
jq accepts a filename argument, and it would be better to use that than change whitelist to include a call to cat:
jq -r 'MYFILTER' "$whitelist"
Using filters such as .[].evidence' makes your script very fragile, as it assumes the top-level array will always have just one entry. If you want just the first entry in the array, you could specify that, though maybe using first would be even more robust:
first(.[].evidence)
Although jq is fairly light-weight, it is worth noting that all those calls to jq could be reduced to one, though of course your shell script would then have to be modified as well. See e.g.
Attempting to assign multiple variables using jq
Try changing
whitelist=whitelist_tmp.json
to
whitelist="cat whitelist_tmp.json"
I think the error was occurring because shell was trying & failing to execute "whitelist_tmp.json" as its own command/function instead of piping the contents of the file to jq.
You also want to add the -r option to jq in the variables you are going to pass in curl, like so:
ruleId=$($whitelist | jq -r '.[].ruleId')
alert=$($whitelist | jq -r '.[].alertName')
newLevel=$($whitelist | jq -r '.[].newLevel')
url=$($whitelist | jq -r '.[].url')
urlIsRegex=$($whitelist | jq -r '.[].urlIsRegex')
enabled=$($whitelist | jq -r '.[].enabled')
parameter=$($whitelist | jq -r '.[].parameter')
evidence=$($whitelist | jq -r '.[].evidence')
Without the -r option, any strings that jq outputs will be quoted, which could be escaping your script's quotation in the curl command. The -r option tells jq to output the raw string data without any quotation.

Hiding sensitive ruby shell commands

I'm using fastlane and sh command to decrypt some credentials but seems ruby prints the output in logs. How do I hide the sensitive information from logs?
cmd_decrypt = "openssl enc -aes-256-cbc -d -a -k \"#{ENV["MATCH_PASSWORD"]}\" -in #{enc_file} -out #{dec_file[0]}"
sh(cmd_decrypt)
output:
[09:38:15]: --------------------------------------------------------------------
[09:38:15]: Step: openssl enc -aes-256-cbc -d -a -k "PASSWORD_SHOWN!" -in /var/folders/7g/yy/T/d20190925-1304-1qv6cj1/vault/zz-out /var/folders/7g/yy/T/d20190925-1304-1qv6cj1/vault/xx
[09:38:15]: --------------------------------------------------------------------
[09:38:15]: $ openssl enc -aes-256-cbc -d -a -k "PASSWORD_SHOWN!" -in /var/folders/7g/yy/T/d20190925-1304-1qv6cj1/vault/zz -out /var/folders/7g/yy/T/d20190925-1304-1qv6cj1/vault/xx
You can pass sh extra parameters. In this case, you would call it like this:
sh(cmd_decrypt, log: false)
The documentation for sh is here: https://docs.fastlane.tools/actions/sh/
You get can get the docs for other built-in actions here:
https://docs.fastlane.tools/actions/
And the docs for other plugin's actions here: https://docs.fastlane.tools/plugins/available-plugins/
Since you have an environment variable, why not just run with that?
cmd_decrypt = "openssl enc -aes-256-cbc -d -a -k \"$MATCH_PASSWORD\" -in #{enc_file} -out #{dec_file[0]}"
sh(cmd_decrypt)
From there shell interpolation should take over and make it work. One thing to note is your -in parameter doesn't have shell escaping, which it usually must have, done using shellescape.
You really should be specifying these as separate arguments, though, whenever possible to avoid injection issues. The problem is you lose shell interpolation at that point.
The good news is you can always write a wrapper script to provide safety and ease of use, something like:
#!/bin/sh
# descrypt.sh
openssl enc -aes-256-cbc -d -a -k "$MATCH_PASSWORD" -in $1 -out $2
So then you can call it like this:
sh('descrypt.sh', enc_file, dec_file[0])
Now it logs something a lot quieter as well as a bonus. You can pick which arguments to pass through, or even throw them all through with $*.

Why is this bash/CURL call to REST services giving inconsistent results with parameters?

I have written a smoke-testing script that uses BASH script & Curl to test RESTful web services we're working on. The script reads a file, and interprets each line as a URL suffix and parameters for a Curl REST call.
Unfortunately, the script gives unexpected results when I adapted it to run HTTP POST calls as well as GET calls. It does not give the same results running the command on its own, vs. in script:
The BASH Script:
IFS=$'\n' #Don't split an input URL line at spaces
RESTHOST='hostNameAndPath' #Can't give this out
URL="/activation/v2/activationInfo --header 'Content-Type:Application/xml'"
URL2="/activation/v2/activationInfo"
OUTPUT=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL -d #"./activation_post.txt" -X POST`
echo 'out:' $OUTPUT
OUTPUT2=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL2 --header 'Content-Type:Application/xml' -d #'./activation_post.txt' -X POST`
echo 'out2:' $OUTPUT2
Results Out:
out: 505
out2: 200
So, the first call fails (HTTP return code 505, HTTP Version Not Supported), and the second call succeeds (return code "OK").
Why does the first call fail, and how do I fix it? I've verified they should execute the same command (evaluating in echo). I am sure there is something basic I'm missing, as I am just NOW learning Bash scripting.
I think I have found the problem! It is caused by IFS=$'\n'! Because of this, variable expansion does not work as expected. It does not let to split the arguments specified in the URL string!
As a result the SERVER_PROTOCOL variable on the server side will be set to '--header Content-Type:Application/xml HTTP/1.1' instead of "HTTP/1.1", and the CONTENT_TYPE will be 'application/x-www-form-urlencoded' instead of 'Application/xml'.
To show the root of the problem in detail:
VAR="Solaris East"
printf "+%s+ " $VAR
echo "==="
IFS=$'\n'
printf "+%s+ " $VAR
Output:
+Solaris+ +East+ ===
+Solaris East+
So the $VAR expansion does not work as expected because of IFS=$'\n'!
Solution: Do not use IFS=$'\n' and replace space to %20 in URL!
URL=${URL2// /%20}" --header Content-Type:Application/xml"
In this case your first curl call will work properly!
If You still use IFS=$'\n' and give --header option in the command line it will not work properly if URL contains a space, because the server will fail to process it (I tested on apache)!
Even You still cannot use HEADER="--header Content-Type:Application/xml" as expanding $HEADER will result one(!) argument for curl, namely --header Content-Type:Application/xml instead of splitting them into two.
So I may suggest to replace spaces in URL to %20 anyway!
The single quotes surrounding Content-Type:Application/xml, because they are quoted in the value of URL are treated as literal quotes and not removed when $URL is expanded in that call to curl. As a result, you are passing an invalid HTTP header. Just use
URL="/activation/v2/activationInfo --header Content-Type:Application/xml"
OUTPUT=`curl -sL -m 30 -w "%{http_code}" -o /dev/null $RESTHOST$URL -d #"./activation_post.txt" -X POST`
However, it's not a great idea to rely on word-splitting like this to combine two separate pieces of the call to curl in a single variable. Try something like this instead:
URLPATH="activation/v2/activationInfo"
HEADERS=("--header" "Content-Type:Application/xml")
OUTPUT=$( curl -SL -m 30 -w "%{http_code}" -o /dev/null "$RESTHOST/$URL" "${HEADERS[#]}" -d #'./activation_post.txt' -X POST )

Resources