bash script - unable to set variable with double quotes in value - bash

Need help in fixing this bash script to set a variable with a value including double quotes. Somehow I am defining this incorrectly as my values foo and bar are not enclosed in double quotes as needed.
My script thus far:
#!/usr/local/bin/bash
set -e
set -x
host='127.0.0.1'
db='mydev'
_account="foo"
_profile="bar"
_version=$1
_mongo=$(which mongo);
exp="db.profile_versions_20170420.find({account:${_account}, profile:${_profile}, version:${_version}}).pretty();";
${_mongo} ${host}/${db} --eval "$exp"
set +x
Output shows:
+ host=127.0.0.1
+ db=mydev
+ _account=foo
+ _profile=bar
+ _version=201704112004
++ which mongo
+ _mongo=/usr/local/bin/mongo
+ exp='db.profile_versions_20170420.find({account:foo, profile:bar, version:201704112004}).pretty();'
+ /usr/local/bin/mongo 127.0.0.1/mydev --eval 'db.profile_versions_20170420.find({account:foo, profile:bar, version:201704112004}).pretty();'
MongoDB shell version: 3.2.4
connecting to: 127.0.0.1/mydev
2017-04-22T15:32:55.012-0700 E QUERY [thread1] ReferenceError: foo is not defined :
#(shell eval):1:36
What i need is account:"foo", profile:"bar" to be enclosed in double quotes.

In bash (and other POSIX shells), the following 2 states are equivalent:
_account=foo
_account="foo"
What you want to do is to preserve the quotations, therefore you can do the following:
_account='"foo"'

Since part of what you're doing here is forming JSON, consider using jq -- which will guarantee that it's well-formed, no matter what the values are.
host='127.0.0.1'
db='mydev'
_account="foo"
_profile="bar"
_version=$1
json=$(jq -n --arg account "$_account" --arg profile "$_profile" --arg version "$_version" \
'{$account, $profile, version: $version | tonumber}')
exp="db.profile_versions_20170420.find($json).pretty();"
mongo "${host}/${db}" --eval "$exp"
This makes jq responsible for adding literal quotes where appropriate, and will avoid attempted injection attacks (for instance, via a version passed in $1 containing something like 1, "other_argument": "malicious_value"), by replacing any literal " in a string with \"; a literal newline with \n, etc -- or, with the | tonumber conversion, failing outright with any non-numeric value.
Note that some of the syntax above requires jq 1.5 -- if you have 1.4 or prior, you'll want to write {account: $account, profile: $profile} instead of being able to write {$account, $profile} with the key names inferred from the variable names.

When you need to use double quotes inside a double quoted string, escape them with backslashes:
$ foo="acount:\"foo\"" sh -c 'echo $foo'
acount:"foo"

I needed to enquote something already in a variable and stick that in a variable. Expanding on Robert Seaman's answer, I found this worked:
VAR='"'$1'"'
(single quote, double quote, single quote, variable,single quote, double quote, single quote)

Related

How to do a bash `for` loop in terraform termplatefile?

I'm trying to include a bash script in an AWS SSM Document, via the Terraform templatefile function. In the aws:runShellScript section of the SSM document, I have a Bash for loop with an # sign that seems to be creating an error during terraform validate.
Version of terraform: 0.13.5
Inside main.tf file:
resource "aws_ssm_document" "magical_document" {
name = "magical_ssm_doc"
document_type = "Command"
document_format = "YAML"
target_type = "/AWS::EC2::Instance"
content = templatefile(
"${path.module}/ssm-doc.yml",
{
Foo: var.foo
}
)
}
Inside my ssm-doc.yaml file, I loop through an array:
for i in "$\{arr[#]\}"; do
if test -f "$i" ; then
echo "[monitor://$i]" >> $f
echo "disabled=0" >> $f
echo "index=$INDEX" >> $f
fi
done
Error:
Error: Error in function call
Call to function "templatefile" failed:
./ssm-doc.yml:1,18-19: Invalid character;
This character is not used within the language., and 1 other diagnostic(s).
I tried escaping the # symbol, like \#, but it didn't help. How do I
Although the error is pointing to the # symbol as being the cause of the error, it's the ${ } that's causing the problem, because this is Terraform interpolation syntax, and it applies to templatefiles too. As the docs say:
The template syntax is the same as for string templates in the main Terraform language, including interpolation sequences delimited with ${ ... }.
And the way to escape interpolation syntax in Terraform is with a double dollar sign.
for i in "$${arr[#]}"; do
if test -f "$i" ; then
echo "[monitor://$i]" >> $f
echo "disabled=0" >> $f
echo "index=$INDEX" >> $f
fi
done
The interpolation syntax is useful with templatefile if you're trying to pass in an argument, such as, in the question Foo. This argument could be accessed within the yaml file as ${Foo}.
By the way, although this article didn't give the answer to this exact issue, it helped me get a deeper appreciation for all the work Terraform is doing to handle different languages via the templatefile function. It had some cool tricks for doing replacements to escape for different scenarios.

Variable not expanding in Double quotes for bash script

I have a bash script where i'm trying to call a curl which is having a variable value as input. When trying to execute the bash script the variable value is not getting expanded in double quotes.
Expected curl in script after variable expansion should be as following:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id="fgsd34sff334" '
Getting executed as follows when observed in debug mode:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\""$idp_sub"\" '
My bash script is as follows:
#!/bin/bash
idp_sub=""
for idp_sub in $(cat /opt/SP/jboss/home/mayur/es_idp_sub.txt)
do
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\""$idp_sub"\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
done
How does do i expand the variable value within double quotes as shown above in expected output ?
Your double-quoted string is inside single quotes, where it won't be expanded.
Compare:
foo=bar
echo 'foo=\""$foo\"'
echo 'foo="'"$foo"'"'
In the second example, we end the single quotes, and double-quote $foo, then start new single quotes for the final '.
It's probably easier to read if we expand using printf instead:
printf 'foo=%s\n' "$foo"
That's something you might want to run as a process substitution.
BUT...
This is a wrong and dangerous way to construct an SQL query (and the web server is also poor, if it forwards arbitrary queries - I hope it has no write permissions to the data). Read about "SQL command injection" and come back to this code when you understand the issues.
Nothing inside single quotes will be expanded by bash, including any double-quotes, and variable names. The good news is you can end your single-quoted section and immediately start a double-quoted section to introduce the variable, and it will all be concatenated into a single argument for the application (curl). Try:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\"'"$idp_sub"'\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
You can make your code strongly injection-proof by rejecting any string containing a double-quote, but you might reject some strings that have been legitimately escaped.
If you can use the q syntax to quote the string, you can make it more injection-proof, but I guess the attacker just has to inject ]":
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=q\"['"$idp_sub"]'\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
You could then search for and reject the pattern string ]" as your anti-injection, which will allow a much wider class of legitimate strings. You would have to tell the users that you have applied q[] quoting to their input, so they don't have to.

Prevent shell from escaping single quotes

Script:
#!/bin/sh -x
ARGS=""
CMD="./run_this_prog"
. . .
ARGS="-first_args '-A select[val]' "
. . .
$CMD $ARGS
I want the commandline to be expanded like this when I run this shell script:
./run_this_prog -first_args '-A select[val]'
Instead what shell does (note the added '\' before each single quote):
+ ARGS=
+ CMD='./run_this_prog'
+ ARGS='-first_args '\''-A select[val]'\'' '
and what it ran on commandline (escaped every special char - Not what I want):
./run_this_prog -first_args \'\-A select\[val\]\'
I tried escaping single quotes like :
ARGS="-first_args \'-A select[val]\' "
But that resulted in (added '\' after each backslash):
+ ARGS=
+ CMD='./run_this_prog'
+ ARGS='-first_args \'\''-A select[val]\'\'' '
I did my googling but couldn't find anything relevant. What am I missing here?
I am using sh-3.2 on rel6 centOS.
Once a quote is inside a string, it will not work the way you want: Inside a string quotes are not syntactic elements, they are just literal characters. This is one reason why bash offers arrays.
Replace:
#!/bin/sh -x
...
ARGS="-first_args '-A select[val]' "
$CMD $ARGS
With:
#!/bin/bash -x
...
ARGS=(-first_args '-A select[val]')
"$CMD" "${ARGS[#]}"
For a much more detailed discussion of this issue, see: "I'm trying to put a command in a variable, but the complex cases always fail!"

Substituting argument value in bash

I'm unable to substitute the argument value(s) in the bash command as below:
# echo $int1
{"id":"74953939-fd20-4472-8aaa-067e6f4c4106"}
# echo $int2
{"id":"5ef4664d-3600-4df9-a6a9-01ffb0f49422"}
# echo $int3
{"id":"6dc95c01-742e-4225-8298-e5750fe67f27"}
# set -x
# data set net-agent interfaces '["$int1", "$int2", "$int3"]'
+ data set net-agent interfaces '["$int1", "$int2", "$int3"]'
Any idea on why the values are not being substituted?
Thanks!
I'm guessing that the argument to the command should be valid JSON, in which case you should remove the double quotes from around each variable and wrap the entire string in double quotes so variables are expanded:
data set net-agent interfaces "[$int1, $int2, $int3]"
Using set -x, this produces:
$ data set net-agent interfaces "[$int1, $int2, $int3]"
+ data set net-agent interfaces '[{"id":"74953939-fd20-4472-8aaa-067e6f4c4106"}, {"id":"5ef4664d-3600-4df9-a6a9-01ffb0f49422"}, {"id":"6dc95c01-742e-4225-8298-e5750fe67f27"}]'

How to preserve single and double quotes in shell script arguments WITHOUT the ability to control how they pass

I need to receive arguments I have no control over into a shell script, and preserve any single or double quotes. For instance, a script that simply outputs the given arguments should act as follows:
> my_script.sh "double" 'single' none
"double" 'single' none
I don't have the privilege of augmenting the arguments such as in:
> my_script.sh \"double\" \'single\' none
or
> my_script.sh '"double"' "'single'" none
And neither "$#" nor "$*" work.
I also thought of reading from STDIN and try something like:
> echo "double" 'single' none | my_script.sh
thinking it may help, but no breakthrough so far.
Any suggestions?
CSH / PERL solutions are welcomed.
This is not possible (without escaping), because the shell processes the arguments and removes the quotes before your script is called. As a result, your script does not know about the quotes specified on the command line.
You cannot recover the single/double quotes exactly as they were, because the shell 'eats' them. If you need to call some other script from your script, you can e.g. single quote the arguments again. Here is a PERL solution I use:
sub args2shell
{
local (#argv) = #_;
local $" = '\' \'';
local (#margv);
#margv = map { s/'/'\\''/g; $_ } #argv;
return "\'#margv\'" if #margv;
return undef;
}
Example usage:
$args = args2shell #ARGV;
open F, "find -follow $args ! -type d |";
...

Resources