I'm defining a variable as a composition of other variables and some text, and I'm trying to get this variable to not expand its containing variables on the assigning. But I want it to expand when called later. That way I could reuse the same template to print different results as the inner variables keep changing. I'm truing to avoid eval as much as possible as I will be receiving some of the inner variables from third parties, and I do not know what to expect.
My use case, as below, is to have some "calling stack" so I can log all messages with the same format and keep a record of the script, function, and line of the logged message in some format like this: script.sh:this_function:42.
My attempted solution
called.sh:
#!/bin/bash
SCRIPT_NAME="`basename "${BASH_SOURCE[0]}"`"
CURR_STACK="${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"
echo "${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"
echo "${CURR_STACK}"
echo
function _func_1 {
echo "${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"
echo "${CURR_STACK}"
}
_func_1
So, I intend to get the same results while printing the "${CURR_STACK}" as when printing the previous line.
If there is some built-in or other clever way to log this 'call stack', by all means, let me know! I'll gladly wave my code good-bye, but I'd still like to know how to prevent the variables from expanding right away on the assigning of CURR_STACK, but still keep them able to expand further ahead.
Am I missing some shopt?
What I've tried:
Case 1 (expanding on line 4):
CURR_STACK="${SCRIPT_NAME}:${FUNNAME[0]}:${LINENO[0]}"
CURR_STACK="`echo "${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}"`"
CURR_STACK="`echo "\${SCRIPT_NAME}:\${FUNCNAME[0]}:\${LINENO[0]}"`"
called.sh::7 <------------------| These are control lines
called.sh::4 <---------------. .------------| With the results I expect to get.
X
called.sh:_func_1:12 <---´ `-------| Both indicate that the values expanded
called.sh::4 <-------------------------| on line 4 - when CURR_STACK was set.
Case 2 (not expanding at all):
CURR_STACK="\${SCRIPT_NAME}:\${FUNNAME[0]}:\${LINENO[0]}"
CURR_STACK=\${SCRIPT_NAME}:\${FUNCNAME[0]}:\${LINENO[0]}
CURR_STACK="`echo '${SCRIPT_NAME}:${FUNCNAME[0]}:${LINENO[0]}'`"
called.sh::7
${SCRIPT_NAME}:${FUNNAME[0]}:${LINENO[0]} <-------.----| No expansion at all!...
/
called.sh::12 /
${SCRIPT_NAME}:${FUNNAME[0]}:${LINENO[0]} <----´
Shell variables are store plain inert text(*), not executable code; there isn't really any concept of delayed evaluation here. To make something that does something when used, create a function instead of a variable:
print_curr_stack() {
echo "$(basename "${BASH_SOURCE[1]}"):${FUNCNAME[1]}:${BASH_LINENO[0]}"
}
# ...
echo "We are now at $(print_curr_stack)"
# Or just run it directly:
print_curr_stack
Note: using BASH_SOURCE[1] and FUNCNAME[1] gets info about context the function was run from, rather than where it is in the function itself. But for some reason I'm not clear on, BASH_LINENO[1] gets the wrong info, and BASH_LINENO[0] is what you want.
You could also write it to allow the caller to specify additional text to print:
print_curr_stack() {
echo "$#" "$(basename "${BASH_SOURCE[1]}"):${FUNCNAME[1]}:${BASH_LINENO[0]}"
}
# ...
print_curr_stack "We are now at"
(* There's an exception to what I said about variables just contain inert text: some variables -- like $LINENO, $RANDOM, etc -- are handled specially by the shell itself. But you can't create new ones like this except by modifying the shell itself.)
Are you familiar with eval?
$ a=this; b=is; c=a; d=test;
$ e='echo "$a $b $c $d"';
$ eval $e;
this is a test
$ b='is NOT'; # modify one of the variables
$ eval $e;
this is NOT a test
$ f=$(eval $e); # capture the value of the "eval" statement
$ echo $f;
this is NOT a test
I'm writing some shell functions that allow to print stack traces when errors occur. For this I'm using the BASH_LINENO array which contain the line number for each frame. Then I retrieve the line from the file using BASH_SOURCE array and a subprocess like line="$(tail -n+$lineno "$file" | head -n1)".
Anyway, it works well, except when an error occur within an eval. The problem is that the line number corresponds to the line after the expression given to eval has been expanded. Therefore, when I retrieve the line with head and tail, obviously it's now the wrong one, or it's not a line at all (lineno is superior to the number of lines in the file).
So I wonder how I could get the actual expanded line. I looked at the variables provided by Bash, but none seems to help in this case.
Example, script1.sh:
#!/usr/bin/env bash
eval "$(./script2.sh)"
script2.sh:
#!/usr/bin/env bash
echo
echo
echo
echo false
When I hit the false line when executing script1.sh, the line number I get is 4, and the file source I get is script1.sh, so it's wrong.
When the line is out of the file, I could detect it, and print the first previous eval line instead, but it's very hacky and I'm sure there are a few different cases to handle. And if the line is within the file, then I cannot even know if it's the right one or not.
eval is hell :'(
Ideally, the BASH_COMMAND would be an array as well, and I could retrieve the commands from it instead of reading the files.
Another idea I just have would be to force the user to pipe the result of the expression into a command that will compress it on one line. Any ideas how, or programs to do that? A simple join on ";" seems to naive (again, lots of edge cases).
P.S.: sorry for the title, I have difficulty giving a meaningful title to this one :/
Eventually I found a workaround: by overriding the eval command with my own function, I was able to change the way I print the stack trace for errors happening in eval statements.
eval() {
# pre eval logic
command eval "$#"
# post eval logic
}
Anyway, please don't use eval, or if you do, use only one line arguments:
# GOOD: "easy" to deal with
for i in ...; do
eval "$(some command)"
done
# BAD: this will mess up your line numbers
eval "$(for i in ...; do
some command $i
done)"
I'm having a bit of trouble getting this to work/ knowing if its possible. I'm creating a game using little other than bash, this requires a lot of repeated case statements. I am trying to load all the repeated case statements into a variable, then repeat them when necessary to limit the amount of work it will take to update the shared case statements between different scripts.
Here is what I have:
#!/bin/bash
moo="[m][o][o]) echo 'thank you for following instructions' ;;"
test=$(echo "while true ; do
read -p 'type moo: ' case
case $case in
$moo
*) echo 'type moo please'
esac
done")
"$test"
The problem I run into is:
./case.sh: line 13: $'while true ; do\nread -p \'type moo: \' case\ncase in\n[m][o][o]) echo \'thank you for following instructions\' ;;\n*) echo \'type moo please\' ;;\nesac\ndone': command not found
The information in the moo variable will eventually be in a separate script and will be set by invoking it as a function within that script when I finally get a working model.
It looks like this is a workable idea, I've just reached a loss on how to invoke the variable without it acting up. If anyone has any ideas, I would greatly appreciate it.
Thank you in advance!
It doesn't work because the quotes make the variable expansion be treated as a single word.
But it wouldn't work without quotes, either, because the shell doesn't parse the output of variables for syntax like semicolon and newline. Variable expansion is done after that stage of command parsing. The only processing that's done on expanded variables is word-splitting and wildcard matching.
You need to use eval to perform all command parsing:
eval "$test"
Another problem is that the variable $case is being expanded when you assign the variable test, it's not getting the value being read by read. Since the variable doesn't have a value yet, it's being executed as:
case in ...
and this is invalid syntax. You need to escape the $ so it will be passed through literally.
There's also no need for echo, you can simply assign the string directly.
test="while true ; do
read -p 'type moo: ' case
case \$case in
$moo
*) echo 'type moo please'
esac
done"
In a shell script I wrote to test how functions are returning values I came across an odd unexpected behavior. The code below assumes that when entering the function fnttmpfile the first echo statement would print to the console and then the second echo statement would actually return the string to the calling main. Well that's what I assumed, but I was wrong!
#!/bin/sh
fntmpfile() {
TMPFILE=/tmp/$1.$$
echo "This is my temp file dude!"
echo "$TMPFILE"
}
mainname=main
retval=$(fntmpfile "$mainname")
echo "main retval=$retval"
What actually happens is the reverse. The first echo goes to the calling function and the second echo goes to STDOUT. why is this and is there a better way....
main retval=This is my temp file dude!
/tmp/main.19121
The whole reason for this test is because I am writing a shell script to do some database backups and decided to use small functions to do specific things, ya know make it clean instead of spaghetti code. One of the functions I was using was this:
log_to_console() {
# arg1 = calling function name
# arg2 = message to log
printf "$1 - $2\n"
}
The whole problem with this is that the function that was returning a string value is getting the log_to_console output instead depending on the order of things. I guess this is one of those gotcha things about shell scripting that I wasn't aware of.
No, what's happening is that you are running your function, and it outputs two lines to stdout:
This is my temp file dude!
/tmp/main.4059
When you run it $(), bash will intercept the output and store it in the value. The string that is stored in the variable contains the first linebreak (the last one is removed). So what is really in your "retval" variable is the following C-style string:
"This is my temp file dude!\n/tmp/main.4059"
This is not really returning a string (can't do that in a shell script), it's just capturing whatever output your function returns. Which is why it doesn't work. Call your function normally if you want to log to console.
What is the purpose of a command that does nothing, being little more than a comment leader, but is actually a shell builtin in and of itself?
It's slower than inserting a comment into your scripts by about 40% per call, which probably varies greatly depending on the size of the comment. The only possible reasons I can see for it are these:
# poor man's delay function
for ((x=0;x<100000;++x)) ; do : ; done
# inserting comments into string of commands
command ; command ; : we need a comment in here for some reason ; command
# an alias for `true'
while : ; do command ; done
I guess what I'm really looking for is what historical application it might have had.
Historically, Bourne shells didn't have true and false as built-in commands. true was instead simply aliased to :, and false to something like let 0.
: is slightly better than true for portability to ancient Bourne-derived shells. As a simple example, consider having neither the ! pipeline operator nor the || list operator (as was the case for some ancient Bourne shells). This leaves the else clause of the if statement as the only means for branching based on exit status:
if command; then :; else ...; fi
Since if requires a non-empty then clause and comments don't count as non-empty, : serves as a no-op.
Nowadays (that is: in a modern context) you can usually use either : or true. Both are specified by POSIX, and some find true easier to read. However there is one interesting difference: : is a so-called POSIX special built-in, whereas true is a regular built-in.
Special built-ins are required to be built into the shell; Regular built-ins are only "typically" built in, but it isn't strictly guaranteed. There usually shouldn't be a regular program named : with the function of true in PATH of most systems.
Probably the most crucial difference is that with special built-ins, any variable set by the built-in - even in the environment during simple command evaluation - persists after the command completes, as demonstrated here using ksh93:
$ unset x; ( x=hi :; echo "$x" )
hi
$ ( x=hi true; echo "$x" )
$
Note that Zsh ignores this requirement, as does GNU Bash except when operating in POSIX compatibility mode, but all other major "POSIX sh derived" shells observe this including dash, ksh93, and mksh.
Another difference is that regular built-ins must be compatible with exec - demonstrated here using Bash:
$ ( exec : )
-bash: exec: :: not found
$ ( exec true )
$
POSIX also explicitly notes that : may be faster than true, though this is of course an implementation-specific detail.
I use it to easily enable/disable variable commands:
#!/bin/bash
if [[ "$VERBOSE" == "" || "$VERBOSE" == "0" ]]; then
vecho=":" # no "verbose echo"
else
vecho=echo # enable "verbose echo"
fi
$vecho "Verbose echo is ON"
Thus
$ ./vecho
$ VERBOSE=1 ./vecho
Verbose echo is ON
This makes for a clean script. This cannot be done with '#'.
Also,
: >afile
is one of the simplest ways to guarantee that 'afile' exists but is 0 length.
A useful application for : is if you're only interested in using parameter expansions for their side-effects rather than actually passing their result to a command.
In that case, you use the parameter expansion as an argument to either : or false depending upon whether you want an exit status of 0 or 1. An example might be
: "${var:=$1}"
Since : is a builtin, it should be pretty fast.
: can also be for block comment (similar to /* */ in C language). For example, if you want to skip a block of code in your script, you can do this:
: << 'SKIP'
your code block here
SKIP
Two more uses not mentioned in other answers:
Logging
Take this example script:
set -x
: Logging message here
example_command
The first line, set -x, makes the shell print out the command before running it. It's quite a useful construct. The downside is that the usual echo Log message type of statement now prints the message twice. The colon method gets round that. Note that you'll still have to escape special characters just like you would for echo.
Cron job titles
I've seen it being used in cron jobs, like this:
45 10 * * * : Backup for database ; /opt/backup.sh
This is a cron job that runs the script /opt/backup.sh every day at 10:45. The advantage of this technique is that it makes for better looking email subjects when the /opt/backup.sh prints some output.
It's similar to pass in Python.
One use would be to stub out a function until it gets written:
future_function () { :; }
If you'd like to truncate a file to zero bytes, useful for clearing logs, try this:
:> file.log
You could use it in conjunction with backticks (``) to execute a command without displaying its output, like this:
: `some_command`
Of course you could just do some_command > /dev/null, but the :-version is somewhat shorter.
That being said I wouldn't recommend actually doing that as it would just confuse people. It just came to mind as a possible use-case.
It's also useful for polyglot programs:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
~function(){ ... }
This is now both an executable shell-script and a JavaScript program: meaning ./filename.js, sh filename.js, and node filename.js all work.
(Definitely a little bit of a strange usage, but effective nonetheless.)
Some explication, as requested:
Shell-scripts are evaluated line-by-line; and the exec command, when run, terminates the shell and replaces it's process with the resultant command. This means that to the shell, the program looks like this:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
As long as no parameter expansion or aliasing is occurring in the word, any word in a shell-script can be wrapped in quotes without changing its' meaning; this means that ':' is equivalent to : (we've only wrapped it in quotes here to achieve the JavaScript semantics described below)
... and as described above, the first command on the first line is a no-op (it translates to : //, or if you prefer to quote the words, ':' '//'. Notice that the // carries no special meaning here, as it does in JavaScript; it's just a meaningless word that's being thrown away.)
Finally, the second command on the first line (after the semicolon), is the real meat of the program: it's the exec call which replaces the shell-script being invoked, with a Node.js process invoked to evaluate the rest of the script.
Meanwhile, the first line, in JavaScript, parses as a string-literal (':'), and then a comment, which is deleted; thus, to JavaScript, the program looks like this:
':'
~function(){ ... }
Since the string-literal is on a line by itself, it is a no-op statement, and is thus stripped from the program; that means that the entire line is removed, leaving only your program-code (in this example, the function(){ ... } body.)
Self-documenting functions
You can also use : to embed documentation in a function.
Assume you have a library script mylib.sh, providing a variety of functions. You could either source the library (. mylib.sh) and call the functions directly after that (lib_function1 arg1 arg2), or avoid cluttering your namespace and invoke the library with a function argument (mylib.sh lib_function1 arg1 arg2).
Wouldn't it be nice if you could also type mylib.sh --help and get a list of available functions and their usage, without having to manually maintain the function list in the help text?
#!/bin/bash
# all "public" functions must start with this prefix
LIB_PREFIX='lib_'
# "public" library functions
lib_function1() {
: This function does something complicated with two arguments.
:
: Parameters:
: ' arg1 - first argument ($1)'
: ' arg2 - second argument'
:
: Result:
: " it's complicated"
# actual function code starts here
}
lib_function2() {
: Function documentation
# function code here
}
# help function
--help() {
echo MyLib v0.0.1
echo
echo Usage: mylib.sh [function_name [args]]
echo
echo Available functions:
declare -f | sed -n -e '/^'$LIB_PREFIX'/,/^}$/{/\(^'$LIB_PREFIX'\)\|\(^[ \t]*:\)/{
s/^\('$LIB_PREFIX'.*\) ()/\n=== \1 ===/;s/^[ \t]*: \?['\''"]\?/ /;s/['\''"]\?;\?$//;p}}'
}
# main code
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
# the script was executed instead of sourced
# invoke requested function or display help
if [ "$(type -t - "$1" 2>/dev/null)" = function ]; then
"$#"
else
--help
fi
fi
A few comments about the code:
All "public" functions have the same prefix. Only these are meant to be invoked by the user, and to be listed in the help text.
The self-documenting feature relies on the previous point, and uses declare -f to enumerate all available functions, then filters them through sed to only display functions with the appropriate prefix.
It is a good idea to enclose the documentation in single quotes, to prevent undesired expansion and whitespace removal. You'll also need to be careful when using apostrophes/quotes in the text.
You could write code to internalize the library prefix, i.e. the user only has to type mylib.sh function1 and it gets translated internally to lib_function1. This is an exercise left to the reader.
The help function is named "--help". This is a convenient (i.e. lazy) approach that uses the library invoke mechanism to display the help itself, without having to code an extra check for $1. At the same time, it will clutter your namespace if you source the library. If you don't like that, you can either change the name to something like lib_help or actually check the args for --help in the main code and invoke the help function manually.
I saw this usage in a script and thought it was a good substitute for invoking basename within a script.
oldIFS=$IFS
IFS=/
for basetool in $0 ; do : ; done
IFS=$oldIFS
...
this is a replacement for the code: basetool=$(basename $0)
Another way, not yet mentioned here is the initialisation of parameters in infinite while-loops. Below is not the cleanest example, but it serves it's purpose.
#!/usr/bin/env bash
[ "$1" ] && foo=0 && bar="baz"
while : "${foo=2}" "${bar:=qux}"; do
echo "$foo"
(( foo == 3 )) && echo "$bar" && break
(( foo=foo+1 ))
done