With Bash 4.4, I wrote scripts using "declare -n" but today I learned that when I give those to RedHat 7 users, the scripts fail because their BASH is 4.2.
Here's a small example of the problem, I wonder if you can advise me on good method to backport this to BASH 4.2:
#!/bin/bash
pwd=`pwd`
declare -A parms
parms[engine]=\"Sweave\"
parms[verbose]=FALSE
parms[tangle]=TRUE
## builds $parmstring by concatenating key=value pairs
catarr() {
declare -n __p="$1"
for k in "${!__p[#]}"
do parmstring+=", $k=${__p[$k]}"
done
}
parmstring=""
catarr parms
echo ${parmstring[*]}
The output is supposed to be like this:
$ bash bashmre.sh
, engine="Sweave", verbose=FALSE, tangle=TRUE
But on old BASH, RedHat says declare does not allow "-n".
I think what you're looking for can be achieved using eval:
#!/bin/bash
pwd=`pwd`
declare -A parms
parms[engine]=\"Sweave\"
parms[verbose]=FALSE
parms[tangle]=TRUE
## builds $parmstring by concatenating key=value pairs
catarr() {
eval keys=(\"\${!$1[#]}\")
for k in "${keys[#]}"
do
eval val=\${$1[$k]}
parmstring+=", $k=$val"
done
}
parmstring=""
catarr parms
echo ${parmstring[*]}
I tested this on bash 4.2.37 and got the desired output
It's not at all clean, but you can generate and then execute code. To try to keep the generation safe, use printf %q for substituted values.
catarr() {
local eval_str
printf -v eval_str '
for k in "${!%q[#]}"; do
parmstring+=", $k=${%q[$k]}"
done
' "$1" "$1"
eval "$eval_str"
}
I gave +1 to the answers by #Charles Duffy and #shay, but in the end I did not use either answer.
I took the path that Charles suggested in a comment to my original post. I re-wrote the thing so there is one separate function for each of the arrays that needs to be concatenated. In the actual application were we are using this, there are two arrays, "parms" and "myopts" and so I end up with functions named catparms and catmyopts
catparms(){
for k in "${!parms[#]}"
do parmstring+=", $k=${parms[$k]}"
done
}
catmyopts(){
for k in "${!myopts[#]}"
do optstring+=", $k=${myopts[$k]}"
done
}
If I had a lot of arrays floating around, that method would become tedious.
If that happens, I am pretty sure I would go back to the method I used relying on Bash 4.4 and tell the people with old Bash they have to upgrade or forget about it. I would need to learn how to add code in the script that would detect the current Bash version and terminate if it is not 4.3 or higher. I've not done it, but could.
Related
This question already has answers here:
How to use a variable's value as another variable's name in bash [duplicate]
(6 answers)
Closed 5 years ago.
Let's say I have a variable's name stored in another variable:
myvar=123
varname=myvar
Now, I'd like to get 123 by just using $varname variable.
Is there a direct way for that? I found no such bash builtin for lookup by name, so came up with this:
function var { v="\$$1"; eval "echo "$v; }
so
var $varname # gives 123
Which doesn't look too bad in the end, but I'm wondering if I missed something more obvious.
From the man page of bash:
${!varname}
If the first character of parameter is an exclamation point, a level of
variable indirection is introduced. Bash uses the value of the variable formed from the rest of parameter as the name of the variable;
this variable is then expanded and that value is used in the rest of
the substitution, rather than the value of parameter itself. This is
known as indirect expansion.
There isn't a direct Posix-conforming syntax, only a bashism. I usually do this:
eval t="\$$varname"
This will work on any Posix shell, including those systems where bash is the login shell and /bin/sh is something smaller and faster like ash. I like bash and use it for my login shell but I avoid bashisms in command files.
Note: One problem with writing bash-specific scripts is that even if you can count on bash being installed, it could be anywhere on the path. It might be a good idea in that case to use the fully general /usr/bin/env shebang style, but note that this is still not 100% portable and has security issues.
${!varname} should do the trick
$ var="content"
$ myvar=var
$ echo ${!myvar}
content
I usually look at Advance Bash-Scripting Guide when I need to freshen up my Bash skills.
Regarding your question look at Indirect References
Notation is:
Version < 2
\$$var
Version >= 2
${!varname}
# bmuSetIndirectVar()
# TO DOUBLE CHECK THIS COMMENT AND DEMO
# This function is an helper to read indirect variables.
# i.e. get the content of a variable whose name is saved
# within an other variable. Like:
# MYDIR="/tmp"
# WHICHDIR="MYDIR"
# bmuSetIndirectVar "WHICHDIR" "$MYDIR"
#
bmuSetIndirectVar(){
tmpVarName=$1
locVarName=$1
extVarName=$2
#echo "debug Ind Input >$1< >$2<"
eval tmpVarName=\$$extVarName
#echo "debug Ind Output >$tmpVarName< >$extVarName<"
export $locVarName="${tmpVarName}"
}
I am currently using this little function. I am not fully happy with it, and I have seen different solutions on the web (if I could recall I would write them here), but it seems to work. Within these few lines there is already some redundancy and extra data but it was helpful for debugging.
If you want to see it in place, i.e. where I am using it, check:
https://github.com/mariotti/bmu/blob/master/bin/backmeup.shellfunctions.sh
Of course it is not the best solution, but made me going on with the work, in
the hope I can replace it with something a bit more general soon.
This question already has answers here:
A variable modified inside a while loop is not remembered
(8 answers)
Closed 6 years ago.
I want to fill an associative array in bash in somewhat non-trivial setup. I have a pipeline of commands to produce the required input for the array.
Here is a minimal/toy example:
declare -A mapping
seq 10 | while read i; do
key="key_$i"
val="val_$i"
echo "mapping[$key]=$val"
mapping["${key}"]="${val}"
done
echo "${mapping["key_1"]}"
echo "${mapping["key_2"]}"
In this example mapping is changed inside while, but these changes do not propagate into the global namespace. I think this is because while works inside a separate subshell, thus namespaces have diverged.
In order to avoid (what I suggest) the problem with subshells, I came up with the following:
declare -A mapping
while read i; do
key="key_$i"
val="val_$i"
echo "mapping[$key]=$val"
mapping["${key}"]="${val}"
done < <(seq 10)
echo "${mapping["key_1"]}"
echo "${mapping["key_2"]}"
Thus, the generation part explicitly goes into a subshell, while the while loop left at the top-level alone. The construction works.
My questions are: is there any better way to accomplish my goal? And, is my suggestion about subshells correct? If so, why bash uses a subshell in the first case, but not in the second?
EDIT: after little more digging, the question mostly is a duplicate of this one. A good list of options to handle the issue could be found at http://mywiki.wooledge.org/BashFAQ/024
Not sure if this is a better way than your second code snippet, but a way to solve the first one is to use sub shell { ... } right after the pipe:
declare -A mapping
seq 10 | {
while read i; do
key="key_$i"
val="val_$i"
echo "mapping[$key]=$val"
mapping["${key}"]="${val}"
done
echo "${mapping["key_1"]}"
echo "${mapping["key_2"]}"
}
In this example declaring local variable with different name from that of global scope produces no error but when name is the same as global I get:
line 5: !1: unbound variable
code:
set -u
function get_arr {
local myArr2=("${!1}")
echo ${myArr2[*]}
local myArr=("${!1}")
echo ${myArr[*]}
}
myArr=(one two three)
get_arr myArr[#]
Just to make sure we are on the same sheet of paper, here is the version working on Bash 3.2 (works fine quoted or unquoted). You must either have an environment setting or stray characters in your file, of something unrelated to your script causing issues:
#!/bin/bash
set -u
function get_arr {
local myArr2=("${!1}")
echo ${myArr2[*]}
local myArr=("${!1}")
echo ${myArr[*]}
}
myArr=(one two three)
get_arr "myArr[#]"
exit 0
Version
$ bash --version
GNU bash, version 3.2.39(1)-release (i586-suse-linux-gnu)
Copyright (C) 2007 Free Software Foundation, Inc.
Output
$ bash array_indirect_ref.sh
one two three
one two three
Execution
$ bash -x array_indirect_ref.sh
+ set -u
+ myArr=(one two three)
+ get_arr 'myArr[#]'
+ myArr2=("${!1}")
+ local myArr2
+ echo one two three
one two three
+ myArr=("${!1}")
+ local myArr
+ echo one two three
one two three
Update: it appears that how you declare the passed array inside your function affects whether or not shadowed names will work, even in new bash versions.
I have some bash code that used to work, as of last week, but now fails after I updated cygwin to its current code.
~~~~~~~~~~
My cygwin bash version is now 4.3.39:
$ bash --version
GNU bash, version 4.3.39(2)-release (i686-pc-cygwin)
which is the latest.
~~~~~~~~~~
Consider this bash code:
#!/bin/bash
set -e # exit on first failed command
set -u # exit if encounter never set variable
testArrayArg1() {
declare -a argArray=("${!1}")
echo "testArrayArg1: ${argArray[#]}"
}
testArrayArg2() {
declare -a anArray=("${!1}")
echo "testArrayArg2: ${anArray[#]}"
}
anArray=("a" "b" "c")
testArrayArg1 anArray[#]
testArrayArg2 anArray[#]
Note that testArrayArg2 function uses an array name (anArray) which shadows the subsequent variable name in the script.
Also note that the way I pass the array to the function (anArray[#]) and the way that I declare the array in the function (declare -a anArray=("${!1}")) are taken from Ken Bertelson's answer here.
Both functions above used to always work.
Now, after my cygwin/bash update, testArrayArg1 still works but testArrayArg2 which uses a shadowed array name fails:
$ bash t.sh
testArrayArg1: a b c
t.sh: line 11: !1: unbound variable
Anyone know what changed recently in bash to cause this?
~~~~~~~~~~
I can fix this if I change how I declare the array inside the function from declare -a anArray=("${!1}") to your "local" idiom of local anArray=("${!1}").
So, this code
testArrayArg3() {
local anArray=("${!1}")
echo "testArrayArg3: ${anArray[#]}"
}
testArrayArg3 anArray[#]
works:
testArrayArg3: a b c
~~~~~~~~~~
OK, so the local anArray=("${!1}") function array arg declaration idiom seems to work.
This idiom is mentioned in that SO link that I mentioned above in a hidden comment under Ken Bertelson's answer. To see it, click on the "show 3 more" link and check out Mike Q's comment.
Is it every bit as good as the declare -a anArray=("${!1}") idiom, or does it have drawbacks of its own?
I have some critical code that depends on passing arrays to bash functions, so I really need to get this straight.
What is the purpose of a command that does nothing, being little more than a comment leader, but is actually a shell builtin in and of itself?
It's slower than inserting a comment into your scripts by about 40% per call, which probably varies greatly depending on the size of the comment. The only possible reasons I can see for it are these:
# poor man's delay function
for ((x=0;x<100000;++x)) ; do : ; done
# inserting comments into string of commands
command ; command ; : we need a comment in here for some reason ; command
# an alias for `true'
while : ; do command ; done
I guess what I'm really looking for is what historical application it might have had.
Historically, Bourne shells didn't have true and false as built-in commands. true was instead simply aliased to :, and false to something like let 0.
: is slightly better than true for portability to ancient Bourne-derived shells. As a simple example, consider having neither the ! pipeline operator nor the || list operator (as was the case for some ancient Bourne shells). This leaves the else clause of the if statement as the only means for branching based on exit status:
if command; then :; else ...; fi
Since if requires a non-empty then clause and comments don't count as non-empty, : serves as a no-op.
Nowadays (that is: in a modern context) you can usually use either : or true. Both are specified by POSIX, and some find true easier to read. However there is one interesting difference: : is a so-called POSIX special built-in, whereas true is a regular built-in.
Special built-ins are required to be built into the shell; Regular built-ins are only "typically" built in, but it isn't strictly guaranteed. There usually shouldn't be a regular program named : with the function of true in PATH of most systems.
Probably the most crucial difference is that with special built-ins, any variable set by the built-in - even in the environment during simple command evaluation - persists after the command completes, as demonstrated here using ksh93:
$ unset x; ( x=hi :; echo "$x" )
hi
$ ( x=hi true; echo "$x" )
$
Note that Zsh ignores this requirement, as does GNU Bash except when operating in POSIX compatibility mode, but all other major "POSIX sh derived" shells observe this including dash, ksh93, and mksh.
Another difference is that regular built-ins must be compatible with exec - demonstrated here using Bash:
$ ( exec : )
-bash: exec: :: not found
$ ( exec true )
$
POSIX also explicitly notes that : may be faster than true, though this is of course an implementation-specific detail.
I use it to easily enable/disable variable commands:
#!/bin/bash
if [[ "$VERBOSE" == "" || "$VERBOSE" == "0" ]]; then
vecho=":" # no "verbose echo"
else
vecho=echo # enable "verbose echo"
fi
$vecho "Verbose echo is ON"
Thus
$ ./vecho
$ VERBOSE=1 ./vecho
Verbose echo is ON
This makes for a clean script. This cannot be done with '#'.
Also,
: >afile
is one of the simplest ways to guarantee that 'afile' exists but is 0 length.
A useful application for : is if you're only interested in using parameter expansions for their side-effects rather than actually passing their result to a command.
In that case, you use the parameter expansion as an argument to either : or false depending upon whether you want an exit status of 0 or 1. An example might be
: "${var:=$1}"
Since : is a builtin, it should be pretty fast.
: can also be for block comment (similar to /* */ in C language). For example, if you want to skip a block of code in your script, you can do this:
: << 'SKIP'
your code block here
SKIP
Two more uses not mentioned in other answers:
Logging
Take this example script:
set -x
: Logging message here
example_command
The first line, set -x, makes the shell print out the command before running it. It's quite a useful construct. The downside is that the usual echo Log message type of statement now prints the message twice. The colon method gets round that. Note that you'll still have to escape special characters just like you would for echo.
Cron job titles
I've seen it being used in cron jobs, like this:
45 10 * * * : Backup for database ; /opt/backup.sh
This is a cron job that runs the script /opt/backup.sh every day at 10:45. The advantage of this technique is that it makes for better looking email subjects when the /opt/backup.sh prints some output.
It's similar to pass in Python.
One use would be to stub out a function until it gets written:
future_function () { :; }
If you'd like to truncate a file to zero bytes, useful for clearing logs, try this:
:> file.log
You could use it in conjunction with backticks (``) to execute a command without displaying its output, like this:
: `some_command`
Of course you could just do some_command > /dev/null, but the :-version is somewhat shorter.
That being said I wouldn't recommend actually doing that as it would just confuse people. It just came to mind as a possible use-case.
It's also useful for polyglot programs:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
~function(){ ... }
This is now both an executable shell-script and a JavaScript program: meaning ./filename.js, sh filename.js, and node filename.js all work.
(Definitely a little bit of a strange usage, but effective nonetheless.)
Some explication, as requested:
Shell-scripts are evaluated line-by-line; and the exec command, when run, terminates the shell and replaces it's process with the resultant command. This means that to the shell, the program looks like this:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
As long as no parameter expansion or aliasing is occurring in the word, any word in a shell-script can be wrapped in quotes without changing its' meaning; this means that ':' is equivalent to : (we've only wrapped it in quotes here to achieve the JavaScript semantics described below)
... and as described above, the first command on the first line is a no-op (it translates to : //, or if you prefer to quote the words, ':' '//'. Notice that the // carries no special meaning here, as it does in JavaScript; it's just a meaningless word that's being thrown away.)
Finally, the second command on the first line (after the semicolon), is the real meat of the program: it's the exec call which replaces the shell-script being invoked, with a Node.js process invoked to evaluate the rest of the script.
Meanwhile, the first line, in JavaScript, parses as a string-literal (':'), and then a comment, which is deleted; thus, to JavaScript, the program looks like this:
':'
~function(){ ... }
Since the string-literal is on a line by itself, it is a no-op statement, and is thus stripped from the program; that means that the entire line is removed, leaving only your program-code (in this example, the function(){ ... } body.)
Self-documenting functions
You can also use : to embed documentation in a function.
Assume you have a library script mylib.sh, providing a variety of functions. You could either source the library (. mylib.sh) and call the functions directly after that (lib_function1 arg1 arg2), or avoid cluttering your namespace and invoke the library with a function argument (mylib.sh lib_function1 arg1 arg2).
Wouldn't it be nice if you could also type mylib.sh --help and get a list of available functions and their usage, without having to manually maintain the function list in the help text?
#!/bin/bash
# all "public" functions must start with this prefix
LIB_PREFIX='lib_'
# "public" library functions
lib_function1() {
: This function does something complicated with two arguments.
:
: Parameters:
: ' arg1 - first argument ($1)'
: ' arg2 - second argument'
:
: Result:
: " it's complicated"
# actual function code starts here
}
lib_function2() {
: Function documentation
# function code here
}
# help function
--help() {
echo MyLib v0.0.1
echo
echo Usage: mylib.sh [function_name [args]]
echo
echo Available functions:
declare -f | sed -n -e '/^'$LIB_PREFIX'/,/^}$/{/\(^'$LIB_PREFIX'\)\|\(^[ \t]*:\)/{
s/^\('$LIB_PREFIX'.*\) ()/\n=== \1 ===/;s/^[ \t]*: \?['\''"]\?/ /;s/['\''"]\?;\?$//;p}}'
}
# main code
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
# the script was executed instead of sourced
# invoke requested function or display help
if [ "$(type -t - "$1" 2>/dev/null)" = function ]; then
"$#"
else
--help
fi
fi
A few comments about the code:
All "public" functions have the same prefix. Only these are meant to be invoked by the user, and to be listed in the help text.
The self-documenting feature relies on the previous point, and uses declare -f to enumerate all available functions, then filters them through sed to only display functions with the appropriate prefix.
It is a good idea to enclose the documentation in single quotes, to prevent undesired expansion and whitespace removal. You'll also need to be careful when using apostrophes/quotes in the text.
You could write code to internalize the library prefix, i.e. the user only has to type mylib.sh function1 and it gets translated internally to lib_function1. This is an exercise left to the reader.
The help function is named "--help". This is a convenient (i.e. lazy) approach that uses the library invoke mechanism to display the help itself, without having to code an extra check for $1. At the same time, it will clutter your namespace if you source the library. If you don't like that, you can either change the name to something like lib_help or actually check the args for --help in the main code and invoke the help function manually.
I saw this usage in a script and thought it was a good substitute for invoking basename within a script.
oldIFS=$IFS
IFS=/
for basetool in $0 ; do : ; done
IFS=$oldIFS
...
this is a replacement for the code: basetool=$(basename $0)
Another way, not yet mentioned here is the initialisation of parameters in infinite while-loops. Below is not the cleanest example, but it serves it's purpose.
#!/usr/bin/env bash
[ "$1" ] && foo=0 && bar="baz"
while : "${foo=2}" "${bar:=qux}"; do
echo "$foo"
(( foo == 3 )) && echo "$bar" && break
(( foo=foo+1 ))
done
This question already has answers here:
How to use a variable's value as another variable's name in bash [duplicate]
(6 answers)
Closed 5 years ago.
Let's say I have a variable's name stored in another variable:
myvar=123
varname=myvar
Now, I'd like to get 123 by just using $varname variable.
Is there a direct way for that? I found no such bash builtin for lookup by name, so came up with this:
function var { v="\$$1"; eval "echo "$v; }
so
var $varname # gives 123
Which doesn't look too bad in the end, but I'm wondering if I missed something more obvious.
From the man page of bash:
${!varname}
If the first character of parameter is an exclamation point, a level of
variable indirection is introduced. Bash uses the value of the variable formed from the rest of parameter as the name of the variable;
this variable is then expanded and that value is used in the rest of
the substitution, rather than the value of parameter itself. This is
known as indirect expansion.
There isn't a direct Posix-conforming syntax, only a bashism. I usually do this:
eval t="\$$varname"
This will work on any Posix shell, including those systems where bash is the login shell and /bin/sh is something smaller and faster like ash. I like bash and use it for my login shell but I avoid bashisms in command files.
Note: One problem with writing bash-specific scripts is that even if you can count on bash being installed, it could be anywhere on the path. It might be a good idea in that case to use the fully general /usr/bin/env shebang style, but note that this is still not 100% portable and has security issues.
${!varname} should do the trick
$ var="content"
$ myvar=var
$ echo ${!myvar}
content
I usually look at Advance Bash-Scripting Guide when I need to freshen up my Bash skills.
Regarding your question look at Indirect References
Notation is:
Version < 2
\$$var
Version >= 2
${!varname}
# bmuSetIndirectVar()
# TO DOUBLE CHECK THIS COMMENT AND DEMO
# This function is an helper to read indirect variables.
# i.e. get the content of a variable whose name is saved
# within an other variable. Like:
# MYDIR="/tmp"
# WHICHDIR="MYDIR"
# bmuSetIndirectVar "WHICHDIR" "$MYDIR"
#
bmuSetIndirectVar(){
tmpVarName=$1
locVarName=$1
extVarName=$2
#echo "debug Ind Input >$1< >$2<"
eval tmpVarName=\$$extVarName
#echo "debug Ind Output >$tmpVarName< >$extVarName<"
export $locVarName="${tmpVarName}"
}
I am currently using this little function. I am not fully happy with it, and I have seen different solutions on the web (if I could recall I would write them here), but it seems to work. Within these few lines there is already some redundancy and extra data but it was helpful for debugging.
If you want to see it in place, i.e. where I am using it, check:
https://github.com/mariotti/bmu/blob/master/bin/backmeup.shellfunctions.sh
Of course it is not the best solution, but made me going on with the work, in
the hope I can replace it with something a bit more general soon.