Bash completion scripting - getting a "transparent proxy"-like behaviour - bash

I am trying to write a simple Bash completion script for a program that runs its arguments as a command. A good example of this is kind of program is the prime-run script provided by the nvidia-prime package:
#!/bin/bash
__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia "$#"
This script sets a few environment variables, which instructs the prime driver to use the Nvidia dGPU on a hybrid system. The first argument is treated as the command, and all trailing arguments are passed through. So for example you can run prime-run code . and VSCode will start in the current directory using the dGPU.
Therefore from a completion-script POV, what we want is to basically try to complete as if the prime-run token isn't there (hence "transparent proxy"-like behaviour). To give a rather contrived example:
> prime-run journalc<TAB>
(completes journalctl)
> prime-run journalctl --us<TAB>
(completes --user)
However I am finding this surprisingly difficult in Bash (not that I know how in other shells). So the question is simple: is it possible and if so how?
Ideas I've (hopelessly) had
The simple complete -A command prime-run: the first argument gets completed as a command as expected (let's call it foo), but the following arguments are also completed as commands rather than as arguments to foo
Use some combination of compgen and complete -p to invoke the completion function of foo, but AFAIK the completion function for all foo is locally defined and thus uncallable

TL;DR
bash-completion provides a function named _command_offset (permalink), which is exactly what I need.
# A meta-command completion function for commands like sudo(8), which need to
# first complete on a command, then complete according to that command's own
# completion definition.
Keep reading if you are interested in how I got here.
So I was daydreaming the other day, when it hit me - doesn't sudo basically have the exact same behaviour I want? So the task became simple - reverse engineer the completion script for sudo. Source available here: permalink.
Turns out, most of the code has to do with completing the various options, so it's safe to simply throw most of it out:
L 8-11, 50-52: Related to sudo's edit mode. Safe to ditch.
L 19-24, 27-39, 43-49: These complete sudo's options. Safe to ditch.
So we're left with this:
_sudo()
{
local cur prev words cword split
_init_completion -s || return
for ((i = 1; i <= cword; i++)); do
if [[ ${words[i]} != -* ]]; then
local PATH=$PATH:/sbin:/usr/sbin:/usr/local/sbin
local root_command=${words[i]}
_command_offset $i
return
fi
done
$split && return
} &&
complete -F _sudo sudo sudoedit
The for and if block are there to deal with sudo's options that precede the "guest command". Safe to ditch (after replacing all $i with 1).
The variable $split is only referenced in _init_completion (permalink), and it seems to be used for handling different argument styles (--foo=bar v.s. --foo bar). Same with the -s flag. Irrelevant.
Appending to $PATH and setting $root_command have to do with privilege escalation. Only relevant to sudo.
So after the dust has cleared, by process of elimination, I ended up with this simple chunk of code:
_my-script()
{
local cur prev words cword
_init_completion || return
_command_offset 1
} && complete -F _my-script my-script
Declaring these four local variables and calling _init_completion is standard for all completion scipts, so really it's as simple as one command. Of course someone had to write the massively-complex _command_offset function so lucky me I guess?
Anyways, thank you for reading the story of me messing around and hopefully this will be helpful to some other person in the future.

Related

Set default options for find on Linux

I often have other filing systems mounted on my Ubuntu machine, and as a result, when I do a find, I have to make sure I include the -mount option (which is the same as the -xdev option) to avoid it searching (often slowly) on those filing systems too. Sometimes, however, I forget; then I wonder why the find is taking so long! What I'd like is a way of making find use -mount all the time.
There doesn't seem to be an environment variable I can use, and there doesn't seem to be something like a .findrc file where I can specify options. And I can't easily use an alias because -mount needs to be after the location(s) I want to search. I could make a bash function that takes the arguments for the search and then inserts -mount after all the locations but before the first switch before passing it to the find command; but before I go to the effort, is there already a way of ensuring find uses -mount every time it's run?
In case it's useful to anyone else, here the bash function I came up with:
find () {
# Make a copy of the arguments so they can be altered
local args=("$#")
# Will start to look at the first (zeroth) element
local i=0
# While element i exists and doesn't start with a "-", increment i
while [[ $i -lt ${#args[#]} && ${args[$i]} != -* ]]; do
let ++i
done
# Insert -mount at position i, which is where the first switch currently is
# (or is the end of the argument list).
args=("${args[#]:0:$i}" '-mount' "${args[#]:$i}")
# Use env to locate the find command in the path, and pass the manipulated
# arguments to it.
env find "${args[#]}"
}

Is lazier (ksh-like) function autoloading possible in bash?

My project (port from ksh) use some directories as autoloadable functions.
In those directories each filenames as the name of a function declared inside the file, sourcing that file to declare (implement) the function. Each directories could be considered a 'package' that augment the bash builtin set via functions. I have about 20 packages, and the number of functions per package can be significant (can reach 30 in some packages).
The bash documentation includes an example implementation of autoloading:
https://www.apt-browse.org/browse/ubuntu/trusty/main/all/bash-doc/4.3-6ubuntu1/file/usr/share/doc/bash/examples/functions/autoload.v2
However, that implementation requires the set of potentially-autoloadable functions to be known (and enumerated) at shell startup time.
Is an implementation that doesn't have that limitation possible?
Well, after asking, my turn to 'give' :) to the SO community. I investigated this auto load feature I need and came up with 2 implementations, I provide one them here, so some may suggest enhancements or point out bugs. I'll post the second one on a second post.
The 2 implementations will runs some test cases, so before presenting the implementations I present the common test cases. We have 2 directories a1/ and a2/ that host function definitions located in file with same name as the function, each dir could be considered a 'package' dir containing functions for this package, and then function in there are namespaced with the package name (dir name), with few exception for the test purpose.
./a1/ac_f3::
function ac_f3
{ echo "In a1 ac_f3() : args=$#"
}
./a1/a1_f1::
function a1_f1
{ echo "In a1_f1() : args=$#"
}
./a1/a1_f2::
function a1_f22
{ echo "In a1_f2() : args=$#"
}
./a2/ac_f3::
function f3
{ echo "In a2 ac_f3() : args=$#"
}
./a2/a2_f1::
function a2_f1
{ echo "In a2_f1() : args=$#"
}
ac_f3 is a function that is not namespaced, and then common to both dir a1/ a2/ yet with different implementation, this is to demonstrate the $FPATH precedence.
a1_f2 is a bogus one it doesn't implement the function a1_f2() and then we must fail gracefully.
a1_f1, a2_f1, simply implement a1_f1() a2_f2(), and must be found and executed.
command_not_found_handle implementation
Thanks Charles for bringing the command_not_found_handle option, because, surely auto loadable function are related to the fact that a 'command' has not been found and then we try to find a auto loadable to load and execute.
But amazingly, the bash shell has an interesting "feature", i.e some undocumented behavior.
Bash doc says.
If the search is unsuccessful, the shell searches for a defined
shell function named command_not_found_handle. If that
function exists, it is invoked with the original command and
the original command's arguments as its arguments, and the
function's exit status becomes the exit status of the shell.
If that function is not defined, the shell prints an error
message and returns an exit status of 127.
This is misleading because here we talk about the command_not_found_handle() function invocation, and then we may infere 'from the shell context' and this is not the case.
In the shell logic, we failed to get an alias, then fail to get a function, then failed to get an 'external to the shell' program, and the shell is already in a sub-shell creation mode, so command_not_found_handle() is invoked but in a subshell. not the shell context. This could be OK, but the 'funny feature' here is that the sub-process created is not clean, its $$ and $PPID are not set correctly, may be this will be fixed one day. To exhibit this bash feature we can do
function command_not_found_handle
{ echo $$ ; sh -c 'echo $PPID'
}
PW$ # In a shell context invocation
PW$ command_not_found_handle
2746
2746
PW$ # In a subshell invocation (via command not found)
PW$ qqq
2746
3090
Back to our autoload feature, this mean we want to install more functions in a shell instance, nothing that can be done in a subshell, so basically command_not_found_handle() is of tiny help and can do nothing beside signal its parent we got entered (then a command was not found), we will exploit this feature in our implementation.
# autoload
# This file must be sourced
# - From your rc files if you need autoloadable fuctions from your
# interactive shell
# - From any script that need autoloadable functions.
#
# The FPATH must be set with a set of dirs/ where to look to find
# file name match the function name to source and execute.
#
# Note that if FPATH is exported, this is a way to export functions to
# script subshells
# Create a default command_not_found_handle if none exist
declare -F command_not_found_handle >/dev/null ||
function command_not_found_handle { ! echo bash: $1 command not found>&2; }
# Rename current command_not_found_handle
_cnf_body=$(declare -f command_not_found_handle | tail -n +2)
eval "function _cnf_prev $_cnf_body"
# Change USR1 to your liking
CNF_SIG=USR1
function autoload
{ declare f=$1 ; shift
declare d s
for d in $(IFS=:; echo $FPATH)
do s=$d/$f
[ -f $s -a -r $s ] &&
{ . $s
declare -F $f >/dev/null ||
{ echo "$s exist but don't define $f" >&2 ; return 127
}
$f "$#" ; return
}
done
_cnf_prev $f "$#"
}
trap 'autoload ${BASH_COMMAND[#]}' $CNF_SIG
function command_not_found_handle
{ kill -$CNF_SIG $$
}
WARNING, if you ever use this 'autoload' file be prepared for bash fix, it may one day reflect the real $$ $PPID, in which case you will need to fix the above snippet with
$PPID instead of $$.
Results.
PW$ . /path/to/autoload
PW$ FPATH=a1:a2
PW$ a1_f1 11a 11b 11c
In a1_f1() : args=11a 11b 11c
PW$ a2_f1 21a 21b 21c
In a2_f1() : args=21a 21b 21c
PW$ a1_f2 12a 12b 12c
a1/a1_f2 exist but don't define a1_f2
PW$ ac_f3 c3a c3b c3c
In a1 ac_f3() : args=c3a c3b c3c
PW$ qqq
Command 'qqq' not found, did you mean:
command 'qrq' from snap qrq (0.3.1)
command 'qrq' from deb qrq
See 'snap info <snapname>' for additional versions.
What we got here is correct, a1_f1() a2_f1() are found, loaded, executed.
a1_f2() is nowhere to be found, despite having a file that could host it.
The qqq invocation display the chaining of the handlers, going function autoload fist, then the ubuntu command-not-found package (if installed) meaning we are not loosing the command_not_found_handle() user experience.
Note there is no 'admin' functions here like adding/removing/reloading functions.
Adding is a matter of setting file in dirs present in $FPATH
Removing is a matter of removing the source file and unset -f the function
Reloading is a matter of editing the source file and unset -f the function.
Reloading function can be pretty neat during development in interactive shell, but all this can be done with a simple
unset -f funcname, so basically you edit your source file. unset the function, then call it, you get the latest. Same may happen in a script daemon, one could implement a signal to the daemon and the trap handler would simply unset a set of functions that would then be reloaded without stopping/restarting the daemon.
Another feature here is that shell 'package' are possible, i.e a source file may implement 'many' functions, some are the external API, other are internal to the package, since all is flat in the shell, function are namespaced, and then each external API functions (albeit documented) can be hard linked to the same file. The first external API used will load all the package functions.
In my project, the documentation is extracted from the packages sources, and then hardlink are inferred and build at this time.
PROs and CONs
PROs
Here we got a light signature in the autoload sourcing, i.e from scripts or from bash rc file (interactive), the define of the autoload() is modest.
It is very dynamic, in the sense that function loading and executing is really deferred until really needed.
CONs
It grabs a signal number, that would not be necessary should the command_not_found_handle() be a real function called from the shell context, this could happen one day.
It is implemented on a bash feature that may move (wrong, $$ $PPID) then need maintenance on the moving target.
Conclusion
This implementation is OK for me (I Don't care loosing SIGUSR1). The ideal solution would be that command_not_found_handle() would be cleanly implemented and then called in the shell context. The a similar implementation would be possible without any signal.
This is a second implementation to avoid the signal usage seen in the previous implementation and the usage of the command_not_found_handle() that seems not completly stable.
autoload::
function autoload
{ local d="$1" && [ "$1" ] && shift && autoload "$#"
local identifier='^[_a-zA-Z][_a-zA-Z0-9]*$'
[ -d "$d" -a -x "$d" ] && cd "$d" &&
{ for f in *
do [[ $f =~ $identifier ]] && alias $f=". $PWD/$f;unalias $f;$f"
done
cd ->/dev/null 2>&1
}
}
autoload $# $(IFS=:; echo $FPATH)
Here again we got to source this autolaod file either in rc file or in scripts.
The usage of FPATH is not really needed (see Notes for more details on FPATH)
So basically the idea is to source the autoload file along with a set of directories to look for.
PW$ . /path/to/autoload a1 a2
PW$ alias | grep 'a[12c]_*'
alias a1_f1='. /home/phi/a1/a1_f1;unalias a1_f1;a1_f1'
alias a1_f2='. /home/phi/a1/a1_f2;unalias a1_f2;a1_f2'
alias a2_f1='. /home/phi/a2/a2_f1;unalias a2_f1;a2_f1'
alias ac_f3='. /home/phi/a1/ac_f3;unalias ac_f3;ac_f3'
PW$ declare -F | grep 'a[12c]_*'
After the autoload sourcing, we got all the alias defined and no functions.
This is a bit heavier than the previous implementation, yet pretty lightweight, alias are not costly to create in the shell, even with hundred of them.
PW$ a1_f1 11a 11b 11c
In a1_f1() : args=11a 11b 11c
PW$ a2_f1 21a 21b 21c
In a2_f1() : args=21a 21b 21c
PW$ alias | grep 'a[12c]_*'
alias a1_f2='. /home/phi/a1/a1_f2;unalias a1_f2;a1_f2'
alias ac_f3='. /home/phi/a1/ac_f3;unalias ac_f3;ac_f3'
PW$ declare -F | grep 'a[12c]_*'
declare -f a1_f1
declare -f a2_f1
Here we see that a1_f1() and a2_f2() are then loaded and executed, they are removed from the alias list and added in the function list.
PW$ a1_f2 12a 12b 12c
a1_f2: command not found
PW$ ac_f3 c3a c3b c3c
In a1 ac_f3() : args=c3a c3b c3c
PW$ qqq
Command 'qqq' not found, did you mean:
command 'qrq' from snap qrq (0.3.1)
command 'qrq' from deb qrq
See 'snap info <snapname>' for additional versions.
Here we see that a1_f2() is not found, not well reported as in the previous implementation.
ac_f3() is the one from a1/ as expected.
qqq still provide the command-not-found distro package result if installed ( normal we didn't mess with command_not_found_handle() )
PROs and CONs
PROs
Not sitting on a bash bug, i.e could live for a while after bash updates.
CONs
A little bit heavier than previous implementation, yet acceptable.
Much simpler, well may be not simpler, but surely shorter than proposed examples in bash documentation, and a bit more lazy, i.e function are loaded only when necessary (not the aliases though)
Multi function 'package' files along with hardlink for external API exposure is less performing, because each external API function (hardlink) will trig a reload of the file, unless the package file is well written removing all the excess aliases after loading.

Possible to get bash input while user is at prompt? (Essentially an event listener)

Old stuff:
Background:
- Ultimate goal is to put a script in my .bash_profile that warns me by changing text color if I'm typing a commit message and it gets too
long (yes I'm aware vim has something like this).
Progress:
- I found the read -n option which led me to write this:
while true; do
# This hits at the 53rd character
read -rn53 input
# I have commit aliased to gc so the if is just checking if I'm writing a commit
if [ "${input:0:2}" = "gc" ]; then
printf "\nMessage getting long"
fi
done
Question:
- However, running this takes the user out of the bash prompt. I need a way to do something like this while at a normal prompt. I can't find
information on anything like this. Does that mean it's not possible?
Or am I just going about it the wrong way?
New progress:
I found the bind -x option which led me to write this:
check_commit() {
if [ "${READLINE_LINE:0:13}" == 'git commit -m' ] && [ ${#READLINE_LINE} -gt 87 ]; then
echo "Commit is $((${#READLINE_LINE} - 87)) characters too long!"
fi
READLINE_LINE="$READLINE_LINE$1"
READLINE_POINT=$(($READLINE_POINT+1))
}
bind -x '"\"": check_commit "\""'
It listens for a double quote and if I'm writing a long commit message tells me how many characters I am over the limit. Also puts the character I typed into the current line since it is eaten by the bind.
New question:
Now I just need a way to put in a regex, character list or at least a variable instead of \" so I can listen on more keys (Yes, I'm aware bind -x probably wasn't intended to be used this way. I can check performance/footprint/stability myself). I tried "$char", "${char}", "$(char)" and a few other things, but none seem to work. What is the correct approach here ?
AFAIK, not possible in a sane way if you want this to happen during your normal prompt (when PROMPT_COMMAND and PS1 are evaluated). That would involved binding a custom compiled readline function for every insert-self and alike.
If you want this to happen in a script using prompt builtin, this is crudely possible with a loop of
read -e -i $(munge_buf $buf) -n $(buf_warn_len $buf) -p $(buf_warning $buf) buf
like commands. This will allow you to create munge_buf() to alter the currently typed text if needed, buf_warn_len() to calculate a new len to warn at (which may be very large if warning was already displayed), and buf_warn_msg() to derive a warning message based upon the buffer.

Customizing bash completion output: each suggestion on a new line

When you type something, you often use bash autocompletion: you start writing a command, for example, and you type TAB to get the rest of the word.
As you have probably noticed, when multiple choices match your command, bash displays them like this :
foobar#myserv:~$ admin-
admin-addrsync admin-adduser admin-delrsync admin-deluser admin-listsvn
admin-addsvn admin-chmod admin-delsvn admin-listrsync
I'm looking for a solution to display each possible solution on a new line, similar to the last column on a ls -l. Ever better, it would be perfect if I could apply a rule like this: "if you find less than 10 suggestions, display them one by line, if more => actual display".
bash prior to version 4.2 doesn't allow any control over the output format of completions, unfortunately.
Bash 4.2+ allows switching to 1-suggestion-per-line output globally, as explained in Grisha Levit's helpful answer, which also links to a clever workaround to achieve a per-completion-function solution.
The following is a tricky workaround for a custom completion.
Solving this problem generically, for all defined completions, would be much harder (if there were a way to invoke readline functions directly, it might be easier, but I haven't found a way to do that).
To test the proof of concept below:
Save to a file and source it (. file) in your interactive shell - this will:
define a command named foo (a shell function)
whose arguments complete based on matching filenames in the current directory.
(When foo is actually invoked, it simply prints its argument in diagnostic form.)
Invoke as:
foo [fileNamePrefix], then press tab:
If between 2 and 9 files in the current directory match, you'll see the desired line-by-line display.
Otherwise (1 match or 10 or more matches), normal completion will occur.
Limitations:
Completion only works properly when applied to the LAST argument on the command line being edited.
When a completion is actually inserted in the command line (once the match is unambiguous), NO space is appended to it (this behavior is required for the workaround).
Redrawing the prompt the first time after printing custom-formatted output may not work properly: Redrawing the command line including the prompt must be simulated and since there is no direct way to obtain an expanded version of the prompt-definition string stored in $PS1, a workaround (inspired by https://stackoverflow.com/a/24006864/45375) is used, which should work in typical cases, but is not foolproof.
Approach:
Defines and assigns a custom completion shell function to the command of interest.
The custom function determines the matches and, if their count is in the desired range, bypasses the normal completion mechanism and creates custom-formatted output.
The custom-formatted output (each match on its own line) is sent directly to the terminal >/dev/tty, and then the prompt and command line are manually "redrawn" to mimic standard completion behavior.
See the comments in the source code for implementation details.
# Define the command (function) for which to establish custom command completion.
# The command simply prints out all its arguments in diagnostic form.
foo() { local a i=0; for a; do echo "\$$((i+=1))=[$a]"; done; }
# Define the completion function that will generate the set of completions
# when <tab> is pressed.
# CAVEAT:
# Only works properly if <tab> is pressed at the END of the command line,
# i.e., if completion is applied to the LAST argument.
_complete_foo() {
local currToken="${COMP_WORDS[COMP_CWORD]}" matches matchCount
# Collect matches, providing the current command-line token as input.
IFS=$'\n' read -d '' -ra matches <<<"$(compgen -A file "$currToken")"
# Count matches.
matchCount=${#matches[#]}
# Output in custom format, depending on the number of matches.
if (( matchCount > 1 && matchCount < 10 )); then
# Output matches in CUSTOM format:
# print the matches line by line, directly to the terminal.
printf '\n%s' "${matches[#]}" >/dev/tty
# !! We actually *must* pass out the current token as the result,
# !! as it will otherwise be *removed* from the redrawn line,
# !! even though $COMP_LINE *includes* that token.
# !! Also, by passing out a nonempty result, we avoid the bell
# !! signal that normally indicates a failed completion.
# !! However, by passing out a single result, a *space* will
# !! be appended to the last token - unless the compspec
# !! (mapping established via `complete`) was defined with
# !! `-o nospace`.
COMPREPLY=( "$currToken" )
# Finally, simulate redrawing the command line.
# Obtain an *expanded version* of `$PS1` using a trick
# inspired by https://stackoverflow.com/a/24006864/45375.
# !! This is NOT foolproof, but hopefully works in most cases.
expandedPrompt=$(PS1="$PS1" debian_chroot="$debian_chroot" "$BASH" --norc -i </dev/null 2>&1 | sed -n '${s/^\(.*\)exit$/\1/p;}')
printf '\n%s%s' "$expandedPrompt" "$COMP_LINE" >/dev/tty
else # Just 1 match or 10 or more matches?
# Perform NORMAL completion: let bash handle it by
# reporting matches via array variable `$COMPREPLY`.
COMPREPLY=( "${matches[#]}" )
fi
}
# Map the completion function (`_complete_foo`) to the command (`foo`).
# `-o nospace` ensures that no space is appended after a completion,
# which is needed for our workaround.
complete -o nospace -F _complete_foo -- foo
bash 4.2+ (and, more generally, applications using readline 6.2+) support this with the use of the completion-display-width variable.
The number of screen columns used to display possible matches when performing completion. The value is ignored if it is less than 0 or greater than the terminal screen width. A value of 0 will cause matches to be displayed one per line. The default value is -1.
Run the following to set the behavior for all completions1 for your current session:
bind 'set completion-display-width 0'
Or modify your ~/.inputrc2 file to have:
set completion-display-width 0
to change the behavior for all new shells.
1 See here for a method for controlling this behavior for individual custom completion functions.
2 The search path for the readline init file is $INPUTRC, ~/.inputrc, /etc/inputrc so modify the file appropriate for you.

What is the purpose of the : (colon) GNU Bash builtin?

What is the purpose of a command that does nothing, being little more than a comment leader, but is actually a shell builtin in and of itself?
It's slower than inserting a comment into your scripts by about 40% per call, which probably varies greatly depending on the size of the comment. The only possible reasons I can see for it are these:
# poor man's delay function
for ((x=0;x<100000;++x)) ; do : ; done
# inserting comments into string of commands
command ; command ; : we need a comment in here for some reason ; command
# an alias for `true'
while : ; do command ; done
I guess what I'm really looking for is what historical application it might have had.
Historically, Bourne shells didn't have true and false as built-in commands. true was instead simply aliased to :, and false to something like let 0.
: is slightly better than true for portability to ancient Bourne-derived shells. As a simple example, consider having neither the ! pipeline operator nor the || list operator (as was the case for some ancient Bourne shells). This leaves the else clause of the if statement as the only means for branching based on exit status:
if command; then :; else ...; fi
Since if requires a non-empty then clause and comments don't count as non-empty, : serves as a no-op.
Nowadays (that is: in a modern context) you can usually use either : or true. Both are specified by POSIX, and some find true easier to read. However there is one interesting difference: : is a so-called POSIX special built-in, whereas true is a regular built-in.
Special built-ins are required to be built into the shell; Regular built-ins are only "typically" built in, but it isn't strictly guaranteed. There usually shouldn't be a regular program named : with the function of true in PATH of most systems.
Probably the most crucial difference is that with special built-ins, any variable set by the built-in - even in the environment during simple command evaluation - persists after the command completes, as demonstrated here using ksh93:
$ unset x; ( x=hi :; echo "$x" )
hi
$ ( x=hi true; echo "$x" )
$
Note that Zsh ignores this requirement, as does GNU Bash except when operating in POSIX compatibility mode, but all other major "POSIX sh derived" shells observe this including dash, ksh93, and mksh.
Another difference is that regular built-ins must be compatible with exec - demonstrated here using Bash:
$ ( exec : )
-bash: exec: :: not found
$ ( exec true )
$
POSIX also explicitly notes that : may be faster than true, though this is of course an implementation-specific detail.
I use it to easily enable/disable variable commands:
#!/bin/bash
if [[ "$VERBOSE" == "" || "$VERBOSE" == "0" ]]; then
vecho=":" # no "verbose echo"
else
vecho=echo # enable "verbose echo"
fi
$vecho "Verbose echo is ON"
Thus
$ ./vecho
$ VERBOSE=1 ./vecho
Verbose echo is ON
This makes for a clean script. This cannot be done with '#'.
Also,
: >afile
is one of the simplest ways to guarantee that 'afile' exists but is 0 length.
A useful application for : is if you're only interested in using parameter expansions for their side-effects rather than actually passing their result to a command.
In that case, you use the parameter expansion as an argument to either : or false depending upon whether you want an exit status of 0 or 1. An example might be
: "${var:=$1}"
Since : is a builtin, it should be pretty fast.
: can also be for block comment (similar to /* */ in C language). For example, if you want to skip a block of code in your script, you can do this:
: << 'SKIP'
your code block here
SKIP
Two more uses not mentioned in other answers:
Logging
Take this example script:
set -x
: Logging message here
example_command
The first line, set -x, makes the shell print out the command before running it. It's quite a useful construct. The downside is that the usual echo Log message type of statement now prints the message twice. The colon method gets round that. Note that you'll still have to escape special characters just like you would for echo.
Cron job titles
I've seen it being used in cron jobs, like this:
45 10 * * * : Backup for database ; /opt/backup.sh
This is a cron job that runs the script /opt/backup.sh every day at 10:45. The advantage of this technique is that it makes for better looking email subjects when the /opt/backup.sh prints some output.
It's similar to pass in Python.
One use would be to stub out a function until it gets written:
future_function () { :; }
If you'd like to truncate a file to zero bytes, useful for clearing logs, try this:
:> file.log
You could use it in conjunction with backticks (``) to execute a command without displaying its output, like this:
: `some_command`
Of course you could just do some_command > /dev/null, but the :-version is somewhat shorter.
That being said I wouldn't recommend actually doing that as it would just confuse people. It just came to mind as a possible use-case.
It's also useful for polyglot programs:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
~function(){ ... }
This is now both an executable shell-script and a JavaScript program: meaning ./filename.js, sh filename.js, and node filename.js all work.
(Definitely a little bit of a strange usage, but effective nonetheless.)
Some explication, as requested:
Shell-scripts are evaluated line-by-line; and the exec command, when run, terminates the shell and replaces it's process with the resultant command. This means that to the shell, the program looks like this:
#!/usr/bin/env sh
':' //; exec "$(command -v node)" "$0" "$#"
As long as no parameter expansion or aliasing is occurring in the word, any word in a shell-script can be wrapped in quotes without changing its' meaning; this means that ':' is equivalent to : (we've only wrapped it in quotes here to achieve the JavaScript semantics described below)
... and as described above, the first command on the first line is a no-op (it translates to : //, or if you prefer to quote the words, ':' '//'. Notice that the // carries no special meaning here, as it does in JavaScript; it's just a meaningless word that's being thrown away.)
Finally, the second command on the first line (after the semicolon), is the real meat of the program: it's the exec call which replaces the shell-script being invoked, with a Node.js process invoked to evaluate the rest of the script.
Meanwhile, the first line, in JavaScript, parses as a string-literal (':'), and then a comment, which is deleted; thus, to JavaScript, the program looks like this:
':'
~function(){ ... }
Since the string-literal is on a line by itself, it is a no-op statement, and is thus stripped from the program; that means that the entire line is removed, leaving only your program-code (in this example, the function(){ ... } body.)
Self-documenting functions
You can also use : to embed documentation in a function.
Assume you have a library script mylib.sh, providing a variety of functions. You could either source the library (. mylib.sh) and call the functions directly after that (lib_function1 arg1 arg2), or avoid cluttering your namespace and invoke the library with a function argument (mylib.sh lib_function1 arg1 arg2).
Wouldn't it be nice if you could also type mylib.sh --help and get a list of available functions and their usage, without having to manually maintain the function list in the help text?
#!/bin/bash
# all "public" functions must start with this prefix
LIB_PREFIX='lib_'
# "public" library functions
lib_function1() {
: This function does something complicated with two arguments.
:
: Parameters:
: ' arg1 - first argument ($1)'
: ' arg2 - second argument'
:
: Result:
: " it's complicated"
# actual function code starts here
}
lib_function2() {
: Function documentation
# function code here
}
# help function
--help() {
echo MyLib v0.0.1
echo
echo Usage: mylib.sh [function_name [args]]
echo
echo Available functions:
declare -f | sed -n -e '/^'$LIB_PREFIX'/,/^}$/{/\(^'$LIB_PREFIX'\)\|\(^[ \t]*:\)/{
s/^\('$LIB_PREFIX'.*\) ()/\n=== \1 ===/;s/^[ \t]*: \?['\''"]\?/ /;s/['\''"]\?;\?$//;p}}'
}
# main code
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
# the script was executed instead of sourced
# invoke requested function or display help
if [ "$(type -t - "$1" 2>/dev/null)" = function ]; then
"$#"
else
--help
fi
fi
A few comments about the code:
All "public" functions have the same prefix. Only these are meant to be invoked by the user, and to be listed in the help text.
The self-documenting feature relies on the previous point, and uses declare -f to enumerate all available functions, then filters them through sed to only display functions with the appropriate prefix.
It is a good idea to enclose the documentation in single quotes, to prevent undesired expansion and whitespace removal. You'll also need to be careful when using apostrophes/quotes in the text.
You could write code to internalize the library prefix, i.e. the user only has to type mylib.sh function1 and it gets translated internally to lib_function1. This is an exercise left to the reader.
The help function is named "--help". This is a convenient (i.e. lazy) approach that uses the library invoke mechanism to display the help itself, without having to code an extra check for $1. At the same time, it will clutter your namespace if you source the library. If you don't like that, you can either change the name to something like lib_help or actually check the args for --help in the main code and invoke the help function manually.
I saw this usage in a script and thought it was a good substitute for invoking basename within a script.
oldIFS=$IFS
IFS=/
for basetool in $0 ; do : ; done
IFS=$oldIFS
...
this is a replacement for the code: basetool=$(basename $0)
Another way, not yet mentioned here is the initialisation of parameters in infinite while-loops. Below is not the cleanest example, but it serves it's purpose.
#!/usr/bin/env bash
[ "$1" ] && foo=0 && bar="baz"
while : "${foo=2}" "${bar:=qux}"; do
echo "$foo"
(( foo == 3 )) && echo "$bar" && break
(( foo=foo+1 ))
done

Resources