Embedding commands in strings - bash

Consider the following:
#!/bin/tcsh
set thing = 'marker:echo "quoted argument"'
set a = `echo "$thing" | sed 's/\([^:]*\):\(.*\)/\1/'`
set b = `echo "$thing" | sed 's/\([^:]*\):\(.*\)/\2/'`
echo $a
echo $b
$b
echo "quoted argument"
This gives
marker
echo "quoted argument"
"quoted argument"
quoted argument
If $b is echo "quoted argument", why does evaluating $b give a different result from echo "quoted argument"?
Since I know tcsh is awful (but it's what I have to use for work), here is the same problem in Bash:
thing='marker:echo "quoted argument"'
a=`echo "$thing" | sed 's/\(.*\):\([^:]*\)/\1/'`
b=`echo "$thing" | sed 's/\(.*\):\([^:]*\)/\2/'`
echo $a
echo $b
$b
echo "quoted argument"
The output is the same. Note that, were I doing this in Bash, I would certainly use a map. I do not have that luxury :). The solution must work in tcsh.
Desired Output
I would like $b to behave just as if I typed the command in myself as I see it:
marker
echo "quoted argument"
quoted argument
quoted argument
This is a follow-up question to Accessing array elements with spaces in TCSH.

Yeah, eval is the "solution" here (well the solution is not to have a command in a string in the first place see http://mywiki.wooledge.org/BashFAQ/050 for more).
The reason you see the quotes when you run $b is because of the order of evaluation of a shell command. The very last thing that the shell does, after all other expansions, is to remote quotes (however it doesn't remove quotes that resulted from any of the expansions).
So when you have b='echo "quoted arguments"' and run $b as the command line what happens is that the variable is expanded so you get echo "quoted arguments" and then that is run as-is.
$ c ()
{
printf 'argc: %s\n' "$#";
printf 'argv: %s\n' "$#"
}
$ b='echo "quoted arguments"'
$ c "quoted arguments"
argc: 1
argv: quoted arguments
$ c $b
argc: 3
argv: echo
argv: "quoted
argv: arguments"
$ c "$b"
argc: 1
argv: echo "quoted arguments"
$ eval c $b
argc: 2
argv: echo
argv: quoted arguments
$ eval c "$b"
argc: 2
argv: echo
argv: quoted arguments

One thing you must keep in mind with command substitution is that each pipe and each command you string together executes within its own subshell. Each time that happens, the shell process the command or string you provide:
If $b is echo "quoted argument", why does evaluating $b give a
different result from echo "quoted argument"?
set thing = 'marker:echo "quoted argument"'
set b = `echo "$thing" | sed 's/\([^:]*\):\(.*\)/\2/'`
echo $b
echo "quoted argument"
In the case of b, you are assigning the return from sed exactly as it is returned to b including the quotes. They become part of b. So echo $b is equivalent to echo '"quoted argument"'. Whereas, your echo "quoted argument" prints the string as the characters contained within the quotes, the shell removing the literal quotes.
Sorry for the initial confusion.

Related

Keep quotes inside variables in heredocs (bash)

I have a variable that I receive from an external service that is in JSON format, and I want to use it inside a HEREDOC. I tried to use jq but got a parse error. This happens because the variable inside the HEREDOC don't have quotes.
Below is an example of what happens:
list="[{\"a\"=\"b\"}]"
echo "out"
echo "$list"
echo "len=${#list}"
echo ""
/bin/bash <<-SHELL
echo "in - 1"
echo "$list"
echo "len=${#list}"
echo ""
list2="$list"
echo "in - 2"
echo "\$list2"
echo "len=\${#list2}"
echo ""
list3="[{\"a\"=\"b\"}]"
echo "in - 3"
echo "\$list3"
echo "len=\${#list3}"
SHELL
And the output:
out
[{"a"="b"}]
len=11
in - 1
[{a=b}]
len=11
in - 2
[{a=b}]
len=7
in - 3
[{"a"="b"}]
len=11
I assume this happens because the external variables are expanded before the HEREDOC is executed, but is there a way to make HEREDOC preserve the quotes in the variable that was generated outside it?
Quote the delimiter, so that no parameter expansions happen in the here document. Then pass your JSON value as an argument, rather than embedding it in the script.
list='[{"a"="b"}]'
echo "out"
echo "$list"
echo "len=${#list}"
echo ""
/bin/bash <<-'SHELL' -s "$list"
echo "in - 1"
echo "$1"
echo "len=${#1}"
SHELL
The -s option allows you to provide arguments to a shell that reads its command from standard input, so that "$list" isn't mistaken for the name of the script to execute.

Why need to add escape character for $? in eval?

I have next:
$ export A=1
$ eval "echo $A; echo $A; lll; echo $?"
The output is:
1
1
-bash: lll: command not found
0
I don't know why I cannot get the exit code of lll, and I happen to try next:
$ export A=1
$ eval "echo $A; echo $A; lll; echo \$?"
1
1
-bash: lll: command not found
127
You can see above works, meanwhile next also work:
$ export A=1
$ eval "echo \$A; echo \$A; lll; echo \$?"
1
1
-bash: lll: command not found
127
I wonder, why I had to add one \ before $?? Also, why \ before $A is not a must?
Remember that, because you have embraced the whole expression to be evaluated, in double quotes. This means that all parameters inside it will be expanded, before eval is invoked. If you don't escape $?, the parameters to be expanded are A and ?. You get the value of A before eval is run (which is what you want), but you would also get the value of ? before eval is run (which is not what you want). The backslash causes a literal $ to be passed to eval, hence defering the time of calculating the status code.
$ eval "echo $A; echo $A; lll; echo $?"
The variables enclosed in double quotes are expanded. The line above is the same as:
$ eval "echo 1; echo 1; lll; echo 0"
In fact, this string is exactly what eval receives as argument. You don't even need to export A for this.
In order to achieve what you want to should enclose the string in single quotes. This way the variables are not expanded any more and eval receives as argument the string exactly as you typed it.
Try these two sets of commands
$ export A=1
$ B="echo $A; echo $A; lll; echo $?"
$ eval $B
$ A=2
$ eval $B
vs.
$ export A=1
$ B='echo $A; echo $A; lll; echo $?'
$ eval $B
$ A=2
$ eval $B
That's a difference, isn't it?

Iterate over a list of quoted strings

I'm trying to run a for loop over a list of strings where some of them are quoted and others are not like so:
STRING='foo "bar_no_space" "baz with space"'
for item in $STRING; do
echo "$item"
done
Expected result:
foo
bar_no_space
baz with space
Actual result:
foo
"bar_no_space"
"baz
with
space"
I can achieve the expected result by running the following command:
bash -c 'for item in '"$STRING"'; do echo "$item"; done;'
I would like to do this without spawning a new bash process or using eval because I do not want to take the risk of having random commands executed.
Please note that I do not control the definition of the STRING variable, I receive it through an environment variable. So I can't write something like:
array=(foo "bar_no_space" "baz with space")
for item in "${array[#]}"; do
echo "$item"
done
If it helps, what I am actually trying to do is split the string as a list of arguments that I can pass to another command.
I have:
STRING='foo "bar_no_space" "baz with space"'
And I want to run:
my-command --arg foo --arg "bar_no_space" --arg "baz with space"
Use an array instead of a normal variable.
arr=(foo "bar_no_space" "baz with space")
To print the values:
print '%s\n' "${arr[#]}"
And to call your command:
my-command --arg "${arr[0]}" --arg "${arr[1]}" --arg "{$arr[2]}"
Can you try something like this:
sh-4.4$ echo $string
foo "bar_no_space" "baz with space"
sh-4.4$ echo $string|awk 'BEGIN{FS="\""}{for(i=1;i<NF;i++)print $i}'|sed '/^ $/d'
foo
bar_no_space
baz with space
Solved: xargs + subshell
A few years late to the party, but...
Malicious Input:
SSH_ORIGINAL_COMMAND='echo "hello world" foo '"'"'bar'"'"'; sudo ls -lah /; say -v Ting-Ting "evil cackle"'
Note: I originally had an rm -rf in there, but then I realized that would be a recipe for disaster when testing variations of the script.
Converted perfectly into safe args:
# DO NOT put IFS= on its own line
IFS=$'\r\n' GLOBIGNORE='*' args=($(echo "$SSH_ORIGINAL_COMMAND" \
| xargs bash -c 'for arg in "$#"; do echo "$arg"; done'))
echo "${args[#]}"
See that you can indeed pass these arguments just like $#:
for arg in "${args[#]}"
do
echo "$arg"
done
Output:
hello world
foo
bar;
sudo
rm
-rf
/;
say
-v
Ting-Ting
evil cackle
I'm too embarrassed to say how much time I spent researching this to figure it out, but once you get the itch... y'know?
Defeating xargs
It is possible to fool xargs by providing escaped quotes:
SSH_ORIGINAL_COMMAND='\"hello world\"'
This can make a literal quote part of the output:
"hello
world"
Or it can cause an error:
SSH_ORIGINAL_COMMAND='\"hello world"'
xargs: unmatched double quote; by default quotes are special to xargs unless you use the -0 option
In either case, it doesn't enable arbitrary execution of code - the parameters are still escaped.
Pure bash parser
Here's a quoted-string parser written in pure bash (what terrible fun)!
Caveat: just like the xargs example above, this errors in the case of an escaped quoted.
Usage
MY_ARGS="foo 'bar baz' qux * "'$(dangerous)'" sudo ls -lah"
# Create array from multi-line string
IFS=$'\r\n' GLOBIGNORE='*' args=($(parseargs "$MY_ARGS"))
# Show each of the arguments array
for arg in "${args[#]}"; do
echo "$arg"
done
Output:
$#: foo bar baz qux *
foo
bar baz
qux
*
Parse Argument Function
Literally going character-by-character and adding to the current string, or adding to the array.
set -u
set -e
# ParseArgs will parse a string that contains quoted strings the same as bash does
# (same as most other *nix shells do). This is secure in the sense that it doesn't do any
# executing or interpreting. However, it also doesn't do any escaping, so you shouldn't pass
# these strings to shells without escaping them.
parseargs() {
notquote="-"
str=$1
declare -a args=()
s=""
# Strip leading space, then trailing space, then end with space.
str="${str## }"
str="${str%% }"
str+=" "
last_quote="${notquote}"
is_space=""
n=$(( ${#str} - 1 ))
for ((i=0;i<=$n;i+=1)); do
c="${str:$i:1}"
# If we're ending a quote, break out and skip this character
if [ "$c" == "$last_quote" ]; then
last_quote=$notquote
continue
fi
# If we're in a quote, count this character
if [ "$last_quote" != "$notquote" ]; then
s+=$c
continue
fi
# If we encounter a quote, enter it and skip this character
if [ "$c" == "'" ] || [ "$c" == '"' ]; then
is_space=""
last_quote=$c
continue
fi
# If it's a space, store the string
re="[[:space:]]+" # must be used as a var, not a literal
if [[ $c =~ $re ]]; then
if [ "0" == "$i" ] || [ -n "$is_space" ]; then
echo continue $i $is_space
continue
fi
is_space="true"
args+=("$s")
s=""
continue
fi
is_space=""
s+="$c"
done
if [ "$last_quote" != "$notquote" ]; then
>&2 echo "error: quote not terminated"
return 1
fi
for arg in "${args[#]}"; do
echo "$arg"
done
return 0
}
I may or may not keep this updated at:
https://git.coolaj86.com/coolaj86/git-scripts/src/branch/master/git-proxy
Seems like a rather stupid thing to do... but I had the itch... oh well.
Here is a way without an array of strings or other difficulties (but with bash calling and eval):
STRING='foo "bar_no_space" "baz with space"'
eval "bash -c 'while [ -n \"\$1\" ]; do echo \$1; shift; done' -- $STRING"
Output:
foo
bar_no_space
baz with space
If You want to do with the strings something more difficult then just echo You can split Your script:
split_qstrings.sh
#!/bin/bash
while [ -n "$1" ]
do
echo "$1"
shift
done
Another part with more difficult processing (capitalizing of a characters for example):
STRING='foo "bar_no_space" "baz with space"'
eval "split_qstrings.sh $STRING" | while read line
do
echo "$line" | sed 's/a/A/g'
done
Output:
foo
bAr_no_spAce
bAz with spAce

Preserve argument splitting when storing command with whitespaces in variable

I'd like to store a command line in a variable, and then execute that command line. The problem is, the command line has arguments with spaces in them. If I do
$ x='command "complex argument"'
$ $x
it calls command with "complex and argument". I tried using "$x" thinking it would preserve the argument splittings, but it only tries to execute a program with the file name command "complex argument". I also tried variations of the quotes (' vs ") and using exec, but it didn't help. Any ideas?
Edit: eval "$x" almost works, but if the whitespaces separating arguments are newlines and not spaces, then it treats the lines as separate commands.
Edit2: The extra " quotes were too much, and made eval interpret the newlines not as spaces, but as command delimiters. The solutions in the answers all work.
For testing purposes, I define:
$ function args() { while [[ "$1" != "" ]]; do echo arg: $1; shift; done }
This works as expected:
$ args "1 2" 3
arg: 1 2
arg: 3
$ x="arg 4 5 6"
$ $x
arg: 4
arg: 5
arg: 6
This doesnt:
$ x="args \"3 4\" 5"
$ $x
arg: "3
arg: 4"
arg: 5
A safe approach is to store your command line in a BASH array:
arr=( command "complex argument" )
Then execute it:
"${arr[#]}"
OR else another approach is to use BASH function:
cmdfunc() {
command "complex argument"
}
cmdfunc
Another solution is
x='command "complex argument"'
eval $x

What is the difference between "$a" and $a in unix [duplicate]

This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Closed 6 years ago.
For example:
#!/bin/sh
a=0
while [ "$a" -lt 10 ]
b="$a"
while [ "$b" -ge 0 ] do
echo -n "$b "
b=`expr $b - 1`
done
echo
a=`expr $a + 1`
done*
The above mentioned script gives the answer in triangle while with out the double quotes, it falls one after the other on diff lines.
After a variable is expanded to its value, word splitting (i.e. separating the value into tokens at whitespace) and filename wildcard expansion takes place unless the variable is inside double quotes.
Example:
var='foo bar'
echo No quotes: $var
echo With quotes: "$var"
will output:
No quotes: foo bar
With quotes: foo bar
Here the difference is how the argument is passed to echo function. Effectively " " will preserve whitespaces.
This:
echo -n "$b "
Is translated to:
echo -n "<number><space>"
While this:
echo -n $b<space>
Will ignore the trailing space and will just output the number:
echo -n <number>
Therefore removing all the spaces that are needed for output to look "triangular".
There are errors in your script:
no do after 1st while
no ; before do after 2nd while
why asterisk on done* at the end?
Now to answer your question.
If used as a paramenter:
"$a" is one argument.
$a (without quotes) is possibly multiple arguments:
Compare:
v='a b'; set $v; echo "\$#=$#, \$1=\"$1\", \$2=\"$2\""
$#=2, $1="a", $2="b"
v='a b'; set "$v"; echo "\$#=$#, \$1=\"$1\", \$2=\"$2\""
$#=1, $1="a b", $2=""

Resources