I'm trying to write a .functions dotfile, with the purpose of loading it (source $HOME/.functions) in my bash, zsh and fish configuration files. I already did it with another (.aliases), successfully. However now I am facing a problem derived from fish not being posix-compliant.
The thing is that aliases share syntax among the three shells, but when it comes to functions fish has its own syntax (function my_func; #code; end instead of function my_func { #code; }). As an example, consider:
Fish:
function say_hello
echo "hello";
end
Bash/Zsh:
say_hello() {
echo "hello";
}
This disables me from just writing them in the file "as is", so I was thinking of writing a conditional such as if [ "$0" = "bash" ] || [ "$0" = "zsh" ]; then #functions_POSIX; else #functions_fish; fi. However, this conditional syntax is also not available in fish!
That's where I'm stuck rn. I would rather not have separate files for each shell.
Thank you in advance.
The only workable answer, in my opinion, is to separate the definitions.
Even if you figure out some way to hack around the fact that fish checks the syntax for the entire file (so wherever you put a bash function definition it will give a syntax error without executing anything), this won't yield a readable file that's nice to edit. You'll just be fighting the hacks.
And function definitions can't be shared anyway, as it's not just a simple search-and-replace of fi to end - the semantics are different, e.g. a command substitution will only be split on newlines, the various special variables ($#) work in different ways, arrays are entirely different, etc...
That means making it a single file isn't workable or helpful, so make your functions scripts instead (if they don't modify the shell's environment) or make a wrapper around a script that does the environment changing, or just write them twice.
I'm trying to get a case statement to match one of four inputs in a Bash/Zsh shell:
v
-v
version
--version
I'm looking for this in the below case statement:
case "$1" in
?(--)version|?(-)v)
# do stuff
;;
esac
And I find that this isn't working. From what I've read, ?(pattern) is how to match 0 or one occurrences of a pattern.
I did have it working by matching the case
--version|version|-v|v)
But it would be nice to have something neater, plus, it's a learning experience!
I imagine this is probably down to me not escaping things properly, but I also tried encompassing my original string in double quotes ("), yet I got no output again.
Any advice?
To provide an answer to this, the best solution (if insisting on using some form of globbing) is:
(|--)version|(|-)v )
But as rightly pointed out by #kvantour, it's much better to just stick with the simpler:
--version|version|-v|v )
For sh/bash/zsh there is https://github.com/koalaman/shellcheck however there won't be support for fish with it https://github.com/koalaman/shellcheck/issues/209 - is there any linters for fish?
To my knowledge, there is not (and obviously this is impossible to prove).
And if someone were to create such a thing, there'd need to be consensus about what the "typical beginner's syntax issues" and "semantic problems that cause a shell to behave strangely and counter-intuitively" are.
Fish doesn't have many of POSIX sh's warts (as it was written as a reaction to them). Some examples from the shellcheck README:
echo $1 # Unquoted variables
Fish's quoting behavior is quite different - in particular, there is no word splitting on variables, so unquoted variables usually do what you want.
v='--verbose="true"'; cmd $v # Literal quotes in variables
This is presumably an (unsuccessful) attempt to defeat word splitting, which isn't necessary.
This example nicely illustrates the issue - there are multiple decades worth of sh scripts. The flaws and unintuitive behaviors are really well known. So well known in fact, that the common-but-incorrect workarounds are known as well. That's just not the case for fish.
(Obviously, other examples do apply to fish as well, especially the "Frequently misused commands" section.)
Some things in fish that I know new users often trip over:
Unquoted variables expand to one argument per element in the list (since every variable is one). That includes zero if the list is empty, which is an issue with test - e.g. test -n $var will return 0 because fish's test builtin is one of the few parts that are POSIX-compatible (since POSIX demands test with one argument returns 0). Double-quote if you always need one argument.
{} expands to nothing and {x} expands to "x", which means find -exec needs quoting, as do some git commit-ishes (HEAD#{4}). (edit: This has since been changed, {} expands to {} and {x} expands to {x} unless x has a comma or other expansion, so HEAD#{4} works)
fish -n or --no-execute "does not execute any commands, only performs syntax checking", so you could do something like what I am doing here:
for f in **/*.fish; do fish -n "$f"; done
I'd like to define a function defvar which defines a variable, e.g.
defvar FOO BAR ## equivalent to FOO=BAR
However, the naive way of defining such a function does not work (presumedly because the shell ends up executing the whole string as a single command?)
defvar () { $1=$2; } ## no dice
One way around this is to explicitly perform the assignment as part of an export call, e.g.
defvar () { export $1=$2; } ## works, but not portable to some older shells
which works in my local bash shell, but it sounds like this may not be portable, and may not work on some older shells.
Is there a portable mechanism for accomplishing this?
We can solve this using eval:
defvar () {
eval "$1='$2'"
}
We construct a statement to evaluate using the function arguments, e.g. "$1='$2'", and then evaluate the constructed string (thereby performing the assignment).
Since eval is defined by the POSIX standard, it's about as portable as it gets. It seems to even work on the heirloom bourne shell, so I imagine this will work on even the older shells out there.
That said, using eval is kind of icky, since someone could write something like:
defvar FOO "'; echo Ouch; '"
That is, it's easy to use that to execute arbitrary code, so either this should only be run with trusted inputs, or sanitized / validated input.
When writing more than a trivial script in bash, I often wonder how to make the code testable.
It is typically hard to write tests for bash code, due to the fact that it is low on functions that take a value and return a value, and high on functions that check and set some aspect in the environment, modify the file-system, invoke a program, etc. - functions that depend on the environment or have side effects. Thus the setup and test code become much more complicated than the code they test.
For example, consider a simple function to test:
function add_to_file() {
local f=$1
cat >> $f
sort -u $f -o $f
}
Test code for this function might consist of:
add_to_file.before:
foo
bar
baz
add_to_file.after:
bar
baz
foo
qux
And test code:
function test_add_to_file() {
cp add_to_file.{before,tmp}
add_to_file add_to_file.tmp
cmp add_to_file.{tmp,after} && echo pass || echo fail
rm add_to_file.tmp
}
Here 5 lines of code are tested by 6 lines of test code and 7 lines of data.
Now consider a slightly more complicated case:
function distribute() {
local file=$1 ; shift
local hosts=( "$#" )
for host in "${hosts[#]}" ; do
rsync -ae ssh $file $host:$file
done
}
I can't even say how to start write a test for that...
So, is there a good way to do TDD in bash scripts, or should I give up and put my efforts elsewhere?
So here is what I learned:
There are some testing frameworks written in bash and for bash,
however...
It is not so much that Bash is not suitable for TDD (although some
other languages come to mind that are a better fit), but the
typical tasks that Bash is used for (Installation, System
configuration), that are hard to write tests for, and in
particularly hard to setup the test.
The poor data structure support in Bash makes it hard to separate
logic from side-effect, and indeed there is typically little logic
in Bash scripts. That makes it hard to break scripts into
testable chunks. There are some functions that can be tested, but
that is the exception, not the rule.
Function are a good thing (tm), but they can only go so far.
Nested functions can be even better, but they are also limited.
At the end of the day, with major effort some coverage can be
obtained, but it will test the less interesting part of the code,
and will keep the bulk of the testing as a good (or bad) old manual
testing.
Meta: I decided to answer (and accept) my own question, because I was unable to choose between Sinan Ünür's (voted up) and mouviciel's (voted up) answers that where equally useful and insightful. I want to note Stefano Borini's answer, that although not impressed me initially, I learned to appreciate it over time. Also his design patterns or best practices for shell scripts answer (voted up) referred above was useful.
If you are writing code at the same time with tests, try to make it high on functions that don't use anything besides their parameters and don't modify environment. That is, if your function might as well run in a subshell, then it will be easy to test. It takes some arguments and outputs something to stdout, or to a file, or maybe it does something on the system, but caller does not feel side effects.
Yes, you will end up with big chain of functions passing down some WORKING_DIR variable that might as well be global, but this is minor inconvenience comparing to the task of tracking what does each function read and modify. Enabling unit tests is just a free bonus too.
Try to minimize cases where you need output. A little subshell abuse will go long way to keeping things nicely separated (at the expense of performance).
Instead of linear structure, where functions are called, set some environment, then other ones are called, all pretty much on one level, try to go for deep call tree with minimum data going back. Returning stuff in bash is inconvenient if you adopt self-imposed abstinence from global vars...
From an implementation point of view, I suggest shUnit2 or bats.
From a practical point of view, I suggest not to give up. I use TDD on bash scripts and I confirm that it is worth the effort.
Of course, I get about twice as many lines of test than of code but with complex scripts, efforts in testing are a good investment. This is true in particular when your client changes its mind near the end of the project and modifies some requirements. Having a regression test suite is a big aid in changing complex bash code.
If you code a bash program large enough to require TDD, you are using the wrong language.
I suggest you to read my previous post on best practices in bash programming, you will probably find something useful to make your bash program testable, but my statement above stays.
Design patterns or best practices for shell scripts
Writing what Meszaros calls consumer tests is hard in any language. Another approach is to verify the behavior of commands such as rsync manually, then write unit tests to prove specific functionality without hitting the network. In this slightly-modified example, $run is used to print the side-effects if the script is run with the keyword "test"
function distribute {
local file=$1 ; shift
for host in $# ; do
$run rsync -ae ssh $file $host:$file
done
}
if [[ $1 == "test" ]]; then
run="echo"
else
distribute schedule.txt $*
exit 0
fi
#
# Built-in self-tests
#
output=$(mktemp)
expected=$(mktemp)
set -e
trap "rm $got $expected" EXIT
distribute schedule.txt login1 login2 > $output
cat << EOF > $expected
rsync -ae ssh schedule.txt login1:schedule.txt
rsync -ae ssh schedule.txt login2:schedule.txt
EOF
diff $output $expected
echo -n '.'
echo; echo "PASS"
You might want to take a look at cucumber/aruba. Did quite a nice job for me.
Additionally, you can stub just about everything you want by doing something like this:
#
# code.sh
#
some_function_calling_some_external_binary()
{
if ! external_binary action_1; then
# ...
fi
if ! external_binary action_2; then
# ...
fi
}
#
# test.sh
#
# now for the test, simply stub your external binary:
external_binary()
{
if [ "$#" = "action_1" ]; then
# stub action_1
elif [ "$#" = "action_2" ]; then
# stub action_2
else
external_binary $#
fi
}
The advanced bash scripting guide has an example of an assert function but here is a simpler and more flexible assert function - just use eval of $* to test any condition.
assert() {
if ! eval $* ; then
echo
echo "===== Assertion failed: \"$*\" ====="
echo "File \"$0\", line:$LINENO line:${BASH_LINENO[*]}"
echo line:$(caller 0)
exit 99
fi
}
# e.g. USAGE:
assert [[ $r == 42 ]]
assert "((r==42))"
BASH_LINENO and caller bash builtin are bash shell specific.
take a look at Outthentic framework - it is designed to create scenarios which runs any Bash code and then analyze the stdout using formal DSL, it's pretty easy to build any Tdd/blackbox tests suite upon this tool.