How to quotes in bash function parameters? - bash

What I'd like to do is take, as an input to a function, a line that may include quotes (single or double) and echo that line exactly as it was provided to the function. For instance:
function doit {
printf "%s " ${#}
eval "${#}"
printf " # [%3d]\n" ${?}
}
Which, given the following input
doit VAR=42
doit echo 'single quote $VAR'
doit echo "double quote $VAR"
Yields the following:
VAR=42 # [ 0]
echo single quote $VAR # [ 0]
echo double quote 42 # [ 0]
So the semantics of the variable expansion are preserved as I'd expect, but I can not get the exact format of the line as it was provided to the function. What I'd like is to have doit echo 'single quote $VAR' result in echo 'single quote $VAR'.
I'm sure this has to do with bash processing the arguments before they are passed to the function; I'm just looking for a way around that (if possible).
Edit
So what I had intended was to shadow the execution of a script while providing an exact replica of the execution that could be used as a diagnostic tool including exit status of each step.
While I can get the desired behavior described above by doing something like
while read line ; do
doit ${line}
done < ${INPUT}
That approach fails in the face of control structures (i.e. if, while, etc). I thought about using set -x but that has it's limitations as well: " becomes ' and exit status is not visible for commands that fail.

I was in a similar position to you in that I needed a script to wrap around an existing command and pass arguments preserving quoting.
I came up with something that doesn't preserve the command line exactly as typed but does pass the arguments correctly and show you what they were.
Here's my script set up to shadow ls:
CMD=ls
PARAMS=""
for PARAM in "$#"
do
PARAMS="${PARAMS} \"${PARAM}\""
done
echo Running: ${CMD} ${PARAMS}
bash -c "${CMD} ${PARAMS}"
echo Exit Code: $?
And this is some sample output:
$ ./shadow.sh missing-file "not a file"
Running: ls "missing-file" "not a file"
ls: missing-file: No such file or directory
ls: not a file: No such file or directory
Exit Code: 1
So as you can see it adds quotes which weren't originally there but it does preserve arguments with spaces in which is what I needed.

The reason this happens is because bash interprets the arguments, as you thought. The quotes simply aren't there any more when it calls the function, so this isn't possible. It worked in DOS because programs could interpret the command line themselves, not that it helps you!

Although #Peter Westlake's answer is correct, and there are no quotes to preserve one can try to deduce if the quotes where required and thus passed in originally. Personally I used this requote function when I needed a proof in my logs that a command ran with the correct quoting:
function requote() {
local res=""
for x in "${#}" ; do
# try to figure out if quoting was required for the $x:
grep -q "[[:space:]]" <<< "$x" && res="${res} '${x}'" || res="${res} ${x}"
done
# remove first space and print:
sed -e 's/^ //' <<< "${res}"
}
And here is how I use it:
CMD=$(requote "${#}")
# ...
echo "${CMD}"

doit echo "'single quote $VAR'"
doit echo '"double quote $VAR"'
Both will work.
bash will only strip the outside set of quotes when entering the function.

Bash will remove the quote when you pass a string with quote in as command line argument. The quote is simply not there anymore when the string is pass to your script. You have no way to know there is a single quote or double quote.
What you probably can do is sth like this:
doit VAR=42
doit echo \'single quote $VAR\'
doit echo \"double quote $VAR\"
In your script you get
echo 'single quote $VAR'
echo "double quote $VAR"
Or do this
doit VAR=42
doit echo 'single quote $VAR'
doit echo '"double quote $VAR"'
In your script you get
echo single quote $VAR
echo "double quote $VAR"

This:
ponerApostrofes1 ()
{
for (( i=1; i<=$#; i++ ));
do
eval VAR="\${$i}";
echo \'"${VAR}"\';
done;
return;
}
As an example has problems when the parameters have apostrophes.
This function:
ponerApostrofes2 ()
{
for ((i=1; i<=$#; i++ ))
do
eval PARAM="\${$i}";
echo -n \'${PARAM//\'/\'\\\'\'}\'' ';
done;
return
}
solves the mentioned problem and you can use parameters including apostrophes inside, like "Porky's", and returns, apparently(?), the same string of parameters when each parameter is quoted; if not, it quotes it. Surprisingly, I don't understand why, if you use it recursively, it doesn't return the same list but each parameter is quoted again. But if you do echo of each one you recover the original parameter.
Example:
$ ponerApostrofes2 'aa aaa' 'bbbb b' 'c'
'aa aaa' 'bbbb b' 'c'
$ ponerApostrofes2 $(ponerApostrofes2 'aa aaa' 'bbbb b' 'c' )
''\''aa' 'aaa'\''' ''\''bbbb' 'b'\''' ''\''c'\'''
And:
$ echo ''\''bbbb' 'b'\'''
'bbbb b'
$ echo ''\''aa' 'aaa'\'''
'aa aaa'
$ echo ''\''c'\'''
'c'
And this one:
ponerApostrofes3 ()
{
for ((i=1; i<=$#; i++ ))
do
eval PARAM="\${$i}";
echo -n ${PARAM//\'/\'\\\'\'} ' ';
done;
return
}
returning one level of quotation less,
doesn't work either, neither alternating both recursively.

If one's shell does not support pattern substitution, i.e. ${param/pattern/string} then the following sed expression can be used to safely quote any string such that it will eval into a single parameter again:
sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/'/"
Combining this with printf it is possible to write a little function that will take any list of strings produced by filename expansion or "$#" and turn it into something that can be safely passed to eval to expand it into arguments for another command while safely preserving parameter separation.
# Usage: quotedlist=$(shell_quote args...)
#
# e.g.: quotedlist=$(shell_quote *.pdf) # filenames with spaces
#
# or: quotedlist=$(shell_quote "$#")
#
# After building up a quoted list, use it by evaling it inside
# double quotes, like this:
#
# eval "set -- $quotedlist"
# for str in "$#"; do
# # fiddle "${str}"
# done
#
# or like this:
#
# eval "\$a_command $quotedlist \$another_parameter"
#
shell_quote()
{
local result=''
local arg
for arg in "$#" ; do
# Append a space to our result, if necessary
#
result=${result}${result:+ }
# Convert each embedded ' to \' , then insert ' at the
# beginning of the line, and append ' at the end of
# the line.
#
result=${result}$(printf "%s\n" "$arg" | \
sed -e "s/'/'\\\\''/g" -e "1s/^/'/" -e "\$s/\$/'/")
done
# use printf(1) instead of echo to avoid weird "echo"
# implementations.
#
printf "%s\n" "$result"
}
It may be easier (and maybe safer, i.e. avoid eval) in some situations to use an "impossible" character as the field separator and then use IFS to control expansion of the value again.

The shell is going to interpret the quotes and the $ before it passes it to your function. There's not a lot your function can do to get the special characters back, because it has no way of knowing (in the double-quote example) whether 42 was hard-coded or if it came from a variable. You will have to escape the special characters if you want them to survive long enough to make it to your function.

Related

Evaluate variable at time of function declaration in shell

I'm setting up my shell environments and I want to be able to use some of the same functions/aliases in zsh as in bash. One of these functions opens either .bashrc or .zshrc in an editor (whichever file is relevant), waits for the editor to close, then reloads the rc file.
# a very simplified version of this function
editrc() {
local rcfile=".$(basename $SHELL)rc"
code -w ~/$rcfile
. ~/$rcfile
}
I use the value of rcfile in a few other functions, so I've pulled it out of the function declaration.
_rc=".$(basename $SHELL)rc"
editrc() {
code -w ~/$_rc
. ~/$_rc
}
# ... other functions that use it ...
unset _rc
However, because I'm a neat freak, I want to unset _rc at the end of my script, but I still want my functions to run correctly. Is there a clever way to evaluate $_rc at the time the function is declared?
I know I could use eval and place everything except $_rc instances within single quotes, but that seems like a pain, since the full version of my function uses both single-quotes and double-quotes.
_rc=".$(basename $SHELL)rc"
eval 'editrc() {
echo Here'"'"'s a thing that uses single quotes. As you can see it'"'"'s a pain.
code -w ~/'$_rc'
. ~/'$_rc'
}'
# ... other functions using `_rc`
unset _rc
I'm guessing I could declare my functions, then do some magic with eval "$(declare -f editrc | awk)". It very well be more pain than it's worth, but I'm always interested in learning new things.
Note: I'd love to generalize this into a utility function that does this.
_myvar=foo
anothervar=bar
myfunc() {
echo $_myvar $anothervar
}
# redeclares myfunc with `$_myvar` expanded, but leaves `$anothervar` as-is
expandfunctionvars myfunc '$_myvar'
Is there a clever way to evaluate $_rc at the time the function is declared?
_rc=".$(basename "$SHELL")rc"
# while you could eval here, source lets you work with a stream
source <(
cat <<EOF
editrc() {
local _rc
# first safely trasfer context
$(declare -p _rc)
EOF
# use quoted here string to do anything inside without caring.
cat <<'EOF'
# do anything else
echo "Here's a thing that uses single quotes. As you can see it's not a pain, just choose proper quoting."
code -w "~/$_rc"
. "~/$_rc"
}
EOF
)
unset _rc
Generally first use declare -p to transfer variables as strings to be evaluated. Then after you "import" variables, use a quoted here document to do anything as in a normal script.
References to read:
<<EOF is a here document. Note the difference in parsing when the here delimiter is quoted vs unquoted.
<(..) is a process substitution
The source command reads a pipe created by process substitution. Inside the process subtitution I output the function to be sourced. With the first here document I output the function name definition, with a local of the variable so that it doesn't pollute global namespace. Then with declare -p I output the variable definition as a properly quoted string later to be sourced by source. Then with a quoted here document I output the rest of the function, so that I do not need to care about quoting.
The code is bash specific, I know nothing about zsh and don't use it.
You could do it with eval too:
eval '
editrc() {
local _rc
# first safely trasfer context
'"$(declare -p _rc)"'
# use quoted here string to do anything inside without caring.
# do anything else
echo "Here'\''s a thing that uses single quotes. As you can see it'\''s not a pain, just choose proper quoting."
code -w "~/$_rc"
. "~/$_rc"
}'
But for me using a quoted here document delimiter allows for easier writing.
While KamilCuck was working on their answer, I devised a function that will take in any function name and a set of variable names, expand just those variables, and redeclare the function.
expandFnVars() {
if [[ $# -lt 2 ]]; then
>&2 echo 'expandFnVars requires at least two arguments: the function name and the variable(s) to be expanded'
return 1
fi
local fn="$1"
shift
local vars=("$#")
if [[ -z "$(declare -F $fn 2> /dev/null)" ]]; then
>&2 echo $fn is not a function.
return 1
fi
foundAllVars=true
for v in $vars; do
if [[ -z "$(declare -p $v 2> /dev/null)" ]]; then
>&2 echo $v is not a declared value.
foundAllVars=false
fi
done
[[ $foundAllVars != true ]] && return 1
fn="$(declare -f $fn)"
for v in $vars; do
local val="$(eval 'echo $'$v)" # get the value of the varable represented by $v
val="${val//\"/\\\"}" # escape any double-quotes
val="${val//\\/\\\\\\}" # escape any backslashes
fn="$(echo "$fn" | sed -r 's/"?\$'$v'"?/"'"$val"'"/g')" # replace instances of "$$v" and $$v with $val
done
eval "$fn"
}
Usage:
foo="foo bar"
bar='$foo'
baz=baz
fn() {
echo $bar $baz
}
expandFnVars fn bar
declare -f fn
# prints:
# fn ()
# {
# echo "$foo" $baz
# }
expandFnVars fn foo
declare -f fn
# prints:
# fn ()
# {
# echo "foo bar" $baz
# }
Looking at it now, I see one flaw. Suppose $bar in the original function was in single-quotes. We probably would not want its value to be replaced. This could be fixed by some clever regex lookbehinds to count the number of unescaped 's, but I'm happy with it as-is.

How to use bash substitution to append a newline at the end of each element of a list

I am looking for a bash one liner that appends a newline after each element of a list. If I call the script as:
./script arg1 arg2 arg3
I want the output to be
arg1
arg2
arg3
I tried different variations of the following. The newline does not get added. Any ordinary char gets added.
# pretty much works except for an extra space
list=${#/%/x}
echo "$list"
# appends 'n'
list=${#/%/\n}
echo "$list"
# appends nothing
list=${#/%/$'\n'}
echo "$list"
# appends nothing, \x078 would append 'x'
list=${#/%/$'\x0D'}
echo "$list"
# appends nothing
CR=$'\n'
list=${#/%/$CR}
echo "$list"
# same issues with arrays
tmp=($#)
list=${tmp/%/\n}
echo "$list"
What fix or alternative do you suggest? I obviously could write a loop or call tr but that's precisely what I thought I could avoid with a bash substitution.
You can use this function with "$#":
f() { printf "%s\n" "$#"; }
f arg1 arg2 arg3
arg1
arg2
arg3
As per man bash
# Expands to the positional parameters, starting from one. When the expansion occurs
within double quotes, each parameter expands to a separate word. That is, "$#" is
equivalent to "$1" "$2" ...
printf would have been my answer as well. Another technique is to use IFS:
$ IFS=$'\n'
$ list="$*"
$ echo "$list"
arg1
arg2
arg3
Notes:
uses ANSI-C quoting for the newline sequence
"$*" (with the quotes, crucial) joins the positional params using the first char of $IFS
quote the shell variable for the echo command to preserve the inner newlines.
That redefines the IFS value for the current shell. You can save the old value first and restore it after:
oldIFS=$IFS; IFS=$'\n'; list="$*"; IFS=$oldIFS
or you can use a subshell so the modification is discarded for you:
$ list=$( IFS=$'\n'; echo "$*" )

Quoting parameters with spaces for later execution

I have this (test) script:
#!/bin/bash
my_cmd_bad_ ( ) {
cmd="$#"
$cmd
}
my_cmd_good_ ( ) {
"$#"
}
my_cmd_bad_ ls -l "file with space"
my_cmd_good_ ls -l "file with space"
The output is (the file does not exist, which is not the point of this question):
ยป ~/test.sh
ls: cannot access file: No such file or directory
ls: cannot access with: No such file or directory
ls: cannot access space: No such file or directory
ls: cannot access file with space: No such file or directory
I am surprised that the first version does not work as expected: the parameter is not quoted, and instead of processing one file, it processes three. Why?
How can I save the command that I want to execute, properly quoted? I need to execute it later, where I do not have "$#" anymore.
A simple rework of this test script would be appreciated.
See similar question: How to pass command line parameters with quotes stored in single variable?
Use those utility functions ho save a command to a string for later execution:
bash_escape() {
# backtick indirection strictly necessary here: we use it to strip the
# trailing newline from sed's output, which Solaris/BSD sed *always* output
# (unlike GNU sed, which outputs "test": printf %s test | sed -e s/dummy//)
out=`echo "$1" | sed -e s/\\'/\\''\\\\'\\'\\'/g`
printf \'%s\' "$out"
}
append_bash_escape() {
printf "%s " "$1"
bash_escape "$2"
}
your_cmd_fixed_ ( ) {
cmd="$#"
while [ $# -gt 0 ] ; do
cmd=`append_bash_escape "$cmd" "$1"` ; shift
done
$cmd
}
You can quote any single parameter and evaluate it later:
my_cmd_bad_ ( ) {
j=0
for i in "$#"; do
cmd["$j"]=\"$"$i"\"
j=$(( $j + 1 ))
done;
eval ${cmd[*]}
}
You are combining three space-delimited strings "ls", "-l", and "file with space" into a single space-delimited string cmd. There's no way to know which spaces were originally quoted (in "file with space") and which spaces were introduced during the assignment to cmd.
Typically, it is not a good idea to try to build up command lines into a single string. Use functions, or isolate the actual command and leave the arguments in $#.
Rewrite the command like this:
my_cmd_bad_ () {
cmd=$1; shift
$cmd "$#"
}
See http://mywiki.wooledge.org/BashFAQ/050
Note that your second version is greatly preferred most of the time. The only exceptions are if you need to do something special. For example, you can't bundle an assignment or redirect or compound command into a parameter list.
The correct way to handle the quoting issue requires non-standard features. Semi-realistic example involving a template:
function myWrapper {
typeset x IFS=$' \t\n'
{ eval "$(</dev/fd/0)"; } <<-EOF
for x in $(printf '%q ' "$#"); do
echo "\$x"
done
EOF
}
myWrapper 'foo bar' $'baz\nbork'
Make sure you understand exactly what's going on here and that you really have a good reason for doing this. It requires ensuring side-effects can't affect the arguments. This specific example doesn't demonstrate a very good use case because everything is hard-coded so you're able to correctly escape things in advance and expand the arguments quoted if you wanted.

Store shell arguments in file while preserving quoting

How can shell arguments be stored in a file for later use while preserving quoting?
To be clear: I don't want to pass on the arguments in place, which could be easily done using "$#". But actually need to store them in a file for later use.
#!/bin/sh
storeargs() {
: #-)
}
if "$1"
then
# useargs is actuall 'git filter-branch'
useargs "$#"
storeargs "$#"
else
# without args use those from previous invocation
eval useargs $(cat store)
fi
.
$ foo 'a "b"' "c 'd'" '\'' 'd
e'
$ foo # behave as if called with same arguments again
The question likely comes down to how to quote a string using common tools in general (awk, perl, ...). I would prefer a solution that does not make the quoted string unreadable. The content of store should look more or less like what I would specify on the commandline.
The question is complicated by the fact that the arguments/strings to be quoted might already contain any kind of valid (shell) quoting and/or any kind of (significant) whitespace, so unconditionally putting single or double quotes around every argument or storing one argument per line won't work.
Why do the heavy lifting?
storeargs() {
while [ $# -gt 0 ]
do
printf "%q " "$1"
shift
done
}
You can now
storeargs "some" "weird $1 \`bunch\` of" params > myparams.txt
storeargs "some" 'weird $1 \`bunch\` of' params >> myparams.txt
cat myparams.txt
Output
some weird\ \ \`bunch\`\ of params
some weird\ \$1\ \\\`bunch\\\`\ of params
This version stores the arguments one per line, so may be a bit ugly in terms of storage. I doubt that it is completely robust, but it satisfies your example (for useargs() { for i in "$#"; do echo $i; done; } ):
storeargs() { printf "%q\n" "$#"; } > store
if test -n "$1"; then
useargs "$#"
storeargs "$#"
else
eval useargs $args
fi
--EDIT--
Use %q in printf to quote the strings (shamelessly copied from sehe's answer). Note that %q is available in the bash built-in printf, but not in standard printf.

Indirect parameter substitution in shell script

I'm having a problem with a shell script (POSIX shell under HP-UX, FWIW). I have a function called print_arg into which I'm passing the name of a parameter as $1. Given the name of the parameter, I then want to print the name and the value of that parameter. However, I keep getting an error. Here's an example of what I'm trying to do:
#!/usr/bin/sh
function print_arg
{
# $1 holds the name of the argument to be shown
arg=$1
# The following line errors off with
# ./test_print.sh[9]: argval=${"$arg"}: The specified substitution is not valid for this command.
argval=${"$arg"}
if [[ $argval != '' ]] ; then
printf "ftp_func: $arg='$argval'\n"
fi
}
COMMAND="XYZ"
print_arg "COMMAND"
I've tried re-writing the offending line every way I can think of. I've consulted the local oracles. I've checked the online "BASH Scripting Guide". And I sharpened up the ol' wavy-bladed knife and scrubbed the altar until it gleamed, but then I discovered that our local supply of virgins has been cut down to, like, nothin'. Drat!
Any advice regarding how to get the value of a parameter whose name is passed into a function as a parameter will be received appreciatively.
You could use eval, though using direct indirection as suggested by SiegeX is probably nicer if you can use bash.
#!/bin/sh
foo=bar
print_arg () {
arg=$1
eval argval=\"\$$arg\"
echo "$argval"
}
print_arg foo
In bash (but not in other sh implementations), indirection is done by: ${!arg}
Input
foo=bar
bar=baz
echo $foo
echo ${!foo}
Output
bar
baz
This worked surprisingly well:
#!/bin/sh
foo=bar
print_arg () {
local line name value
set | \
while read line; do
name=${line%=*} value=${line#*=\'}
if [ "$name" = "$1" ]; then
echo ${value%\'}
fi
done
}
print_arg foo
It has all the POSIX clunkiness, in Bash would be much sorter, but then again, you won't need it because you have ${!}. This -in case it proves solid- would have the advantage of using only builtins and no eval. If I were to construct this function using an external command, it would have to be sed. Would obviate the need for the read loop and the substitutions. Mind that asking for indirections in POSIX without eval, has to be paid with clunkiness! So don't beat me!
Even though the answer's already accepted, here's another method for those who need to preserve newlines and special characters like Escape ( \033 ): Storing the variable in base64.
You need: bc, wc, echo, tail, tr, uuencode, uudecode
Example
#!/bin/sh
#====== Definition =======#
varA="a
b
c"
# uuencode the variable
varB="`echo "$varA" | uuencode -m -`"
# Skip the first line of the uuencode output.
varB="`NUM=\`(echo "$varB"|wc -l|tr -d "\n"; echo -1)|bc \`; echo "$varB" | tail -n $NUM)`"
#====== Access =======#
namevar1=varB
namevar2=varA
echo simple eval:
eval "echo \$$namevar2"
echo simple echo:
echo $varB
echo precise echo:
echo "$varB"
echo echo of base64
eval "echo \$$namevar1"
echo echo of base64 - with updated newlines
eval "echo \$$namevar1 | tr ' ' '\n'"
echo echo of un-based, using sh instead of eval (but could be made with eval, too)
export $namevar1
sh -c "(echo 'begin-base64 644 -'; echo \$$namevar1 | tr ' ' '\n' )|uudecode"
Result
simple eval:
a b c
simple echo:
YQpiCmMK ====
precise echo:
YQpiCmMK
====
echo of base64
YQpiCmMK ====
echo of base64 - with updated newlines
YQpiCmMK
====
echo of un-based, using sh instead of eval (but could be made with eval, too)
a
b
c
Alternative
You also could use the set command and parse it's output; with that, you don't need to treat the variable in a special way before it's accessed.
A safer solution with eval:
v=1
valid_var_name='[[:alpha:]_][[:alnum:]_]*$'
print_arg() {
local arg=$1
if ! expr "$arg" : "$valid_var_name" >/dev/null; then
echo "$0: invalid variable name ($arg)" >&2
exit 1
fi
local argval
eval argval=\$$arg
echo "$argval"
}
print_arg v
print_arg 'v; echo test'
Inspired by the following answer.

Resources