Shell script common template - bash

Question

Millions of developers write shell scripts to solve various types of tasks. I use shell scripts to simplify deployment, life-cycle management, installation or simply as a glue language.
What I've noticed is nobody actually cares about shell scripts style and quality. A lot of teams spend many hours fixing Java, C++, ... style issues, but totally ignore issues in their shell scripts. By the way, usually there is no standard way to implement a shell script within a particular project, so the one may find dozens different, ugly and buggy scripts, spread around the codebase.
To overcome that issue in my projects I decided to create a shell script template, universal and good enough. I will provide my templates as is to make this question a bit more useful. Out of the box these templates provides:
command-line arguments handling
synchronization
some basic help
Arguments handling: getopts (latest version: shell-script-template#github)
#!/bin/bash
# ------------------------------------------------------------------
# [Author] Title
# Description
# ------------------------------------------------------------------
VERSION=0.1.0
SUBJECT=some-unique-id
USAGE="Usage: command -ihv args"
# --- Options processing -------------------------------------------
if [ $# == 0 ] ; then
echo $USAGE
exit 1;
fi
while getopts ":i:vh" optname
do
case "$optname" in
"v")
echo "Version $VERSION"
exit 0;
;;
"i")
echo "-i argument: $OPTARG"
;;
"h")
echo $USAGE
exit 0;
;;
"?")
echo "Unknown option $OPTARG"
exit 0;
;;
":")
echo "No argument value for option $OPTARG"
exit 0;
;;
*)
echo "Unknown error while processing options"
exit 0;
;;
esac
done
shift $(($OPTIND - 1))
param1=$1
param2=$2
# --- Locks -------------------------------------------------------
LOCK_FILE=/tmp/$SUBJECT.lock
if [ -f "$LOCK_FILE" ]; then
echo "Script is already running"
exit
fi
trap "rm -f $LOCK_FILE" EXIT
touch $LOCK_FILE
# --- Body --------------------------------------------------------
# SCRIPT LOGIC GOES HERE
echo $param1
echo $param2
# -----------------------------------------------------------------
Shell Flags (shFlags) allows to simplify command-line arguments handling a lot, so at some moment of time I decided not to ignore such possibility.
Arguments handling: shflags (latest version: shell-script-template#github)
#!/bin/bash
# ------------------------------------------------------------------
# [Author] Title
# Description
#
# This script uses shFlags -- Advanced command-line flag
# library for Unix shell scripts.
# http://code.google.com/p/shflags/
#
# Dependency:
# http://shflags.googlecode.com/svn/trunk/source/1.0/src/shflags
# ------------------------------------------------------------------
VERSION=0.1.0
SUBJECT=some-unique-id
USAGE="Usage: command -hv args"
# --- Option processing --------------------------------------------
if [ $# == 0 ] ; then
echo $USAGE
exit 1;
fi
. ./shflags
DEFINE_string 'aparam' 'adefault' 'First parameter'
DEFINE_string 'bparam' 'bdefault' 'Second parameter'
# parse command line
FLAGS "$#" || exit 1
eval set -- "${FLAGS_ARGV}"
shift $(($OPTIND - 1))
param1=$1
param2=$2
# --- Locks -------------------------------------------------------
LOCK_FILE=/tmp/${SUBJECT}.lock
if [ -f "$LOCK_FILE" ]; then
echo "Script is already running"
exit
fi
trap "rm -f $LOCK_FILE" EXIT
touch $LOCK_FILE
# -- Body ---------------------------------------------------------
# SCRIPT LOGIC GOES HERE
echo "Param A: $FLAGS_aparam"
echo "Param B: $FLAGS_bparam"
echo $param1
echo $param2
# -----------------------------------------------------------------
I do think these templates can be improved to simplify developer's life even more.
So the question is how to improve them to have the following:
built-in logging
better error handling
better portability
smaller footprint
built-in execution time tracking

Answer 1

This is the header of my script shell template (which can be found here: http://www.uxora.com/unix/shell-script/18-shell-script-template).
It is a man look alike which is used to by usage() to diplsay help as well.
#!/bin/ksh
#================================================================
# HEADER
#================================================================
#% SYNOPSIS
#+ ${SCRIPT_NAME} [-hv] [-o[file]] args ...
#%
#% DESCRIPTION
#% This is a script template
#% to start any good shell script.
#%
#% OPTIONS
#% -o [file], --output=[file] Set log file (default=/dev/null)
#% use DEFAULT keyword to autoname file
#% The default value is /dev/null.
#% -t, --timelog Add timestamp to log ("+%y/%m/%d#%H:%M:%S")
#% -x, --ignorelock Ignore if lock file exists
#% -h, --help Print this help
#% -v, --version Print script information
#%
#% EXAMPLES
#% ${SCRIPT_NAME} -o DEFAULT arg1 arg2
#%
#================================================================
#- IMPLEMENTATION
#- version ${SCRIPT_NAME} (www.uxora.com) 0.0.4
#- author Michel VONGVILAY
#- copyright Copyright (c) http://www.uxora.com
#- license GNU General Public License
#- script_id 12345
#-
#================================================================
# HISTORY
# 2015/03/01 : mvongvilay : Script creation
# 2015/04/01 : mvongvilay : Add long options and improvements
#
#================================================================
# DEBUG OPTION
# set -n # Uncomment to check your syntax, without execution.
# set -x # Uncomment to debug this shell script
#
#================================================================
# END_OF_HEADER
#================================================================
And here is the usage functions to go with:
#== needed variables ==#
SCRIPT_HEADSIZE=$(head -200 ${0} |grep -n "^# END_OF_HEADER" | cut -f1 -d:)
SCRIPT_NAME="$(basename ${0})"
#== usage functions ==#
usage() { printf "Usage: "; head -${SCRIPT_HEADSIZE:-99} ${0} | grep -e "^#+" | sed -e "s/^#+[ ]*//g" -e "s/\${SCRIPT_NAME}/${SCRIPT_NAME}/g" ; }
usagefull() { head -${SCRIPT_HEADSIZE:-99} ${0} | grep -e "^#[%+-]" | sed -e "s/^#[%+-]//g" -e "s/\${SCRIPT_NAME}/${SCRIPT_NAME}/g" ; }
scriptinfo() { head -${SCRIPT_HEADSIZE:-99} ${0} | grep -e "^#-" | sed -e "s/^#-//g" -e "s/\${SCRIPT_NAME}/${SCRIPT_NAME}/g"; }
Here is what you should obtain:
# Display help
$ ./template.sh --help
SYNOPSIS
template.sh [-hv] [-o[file]] args ...
DESCRIPTION
This is a script template
to start any good shell script.
OPTIONS
-o [file], --output=[file] Set log file (default=/dev/null)
use DEFAULT keyword to autoname file
The default value is /dev/null.
-t, --timelog Add timestamp to log ("+%y/%m/%d#%H:%M:%S")
-x, --ignorelock Ignore if lock file exists
-h, --help Print this help
-v, --version Print script information
EXAMPLES
template.sh -o DEFAULT arg1 arg2
IMPLEMENTATION
version template.sh (www.uxora.com) 0.0.4
author Michel VONGVILAY
copyright Copyright (c) http://www.uxora.com
license GNU General Public License
script_id 12345
# Display version info
$ ./template.sh -v
IMPLEMENTATION
version template.sh (www.uxora.com) 0.0.4
author Michel VONGVILAY
copyright Copyright (c) http://www.uxora.com
license GNU General Public License
script_id 12345
You can get the full script template here: http://www.uxora.com/unix/shell-script/18-shell-script-template

Answer 2

I would steer clear of relying on bash as the shell and model your solution on top of shell syntax defined by POSIX and use /bin/sh on the shebang. We had a number of surprises recently when Ubuntu changed /bin/sh to dash.
Another pandemic in the shell world is a general misunderstanding of exit status codes. Exiting with an understandable code is what lets other shell scripts programmatically react to specific failures. Unfortunately, there is not a lot of guidance on this beyond the "sysexits.h" header file.
If you are looking for more information about good shell scripting practices, concentrate on Korn shell scripting resources. Ksh programming tends to focus on really programming as opposed to writing haphazard scripts.
Personally, I haven't found much use for shell templates. The unfortunate truth is that most engineers will simply copy and paste your template and continue to write the same sloppy shell code. A better approach is to create a library of shell functions with well-defined semantics and then convince others to use them. This approach will also help with change control. For example, if you find a defect in a template, then every script that was based on it is broken and would require modifications. Using a library makes it possible to fix defects in one place.
Welcome to the world of shell scripting. Writing shell scripts is a bit of a lost art that seems to be entering a renaissance. There were some good books written on the subject in the late 90's - UNIX Shell Programming by Burns and Arthur comes to mind though the Amazon reviews for the book make it seem awful. IMHO, effective shell code embraces the UNIX philosophy as described by Eric S. Raymond in The Art of Unix Programming.

Answer 3

Here's my bash boilerplate with some sane options explained in the comments
#!/usr/bin/env bash
set -e # Abort script at first error, when a command exits with non-zero status (except in until or while loops, if-tests, list constructs)
set -u # Attempt to use undefined variable outputs error message, and forces an exit
set -x # Similar to verbose mode (-v), but expands commands
set -o pipefail # Causes a pipeline to return the exit status of the last command in the pipe that returned a non-zero return value.

Answer 4

If you're concerned about portability, do not use == in tests. Use = instead. Do not explicitly check if $# is 0. Instead, use ${n?error message} the first time you reference a required argument (eg ${3?error message}). This prevents the extremely annoying practice of emitting a usage statement instead of an error message. And most importantly, always put error messages on the right stream and exit with the correct status. For example:
echo "Unknown error while processing options" >&2
exit 1;
It is often convenient to do something like:
die() { echo "$*"; exit 1; } >&2

Answer 5

I will also share my results. The idea behind all these examples is to encourage overall quality. It is also important to make sure final result is safe enough.
Logging
It is really crucial to have proper logging available from the same beginning. I'm just trying to think about production usage.
TAG="foo"
LOG_FILE="example.log"
function log() {
if [ $HIDE_LOG ]; then
echo -e "[$TAG] $#" >> $LOG_FILE
else
echo "[`date +"%Y/%m/%d:%H:%M:%S %z"`] [$TAG] $#" | tee -a $LOG_FILE
fi
}
log "[I] service start"
log "[D] debug message"
Command test
This is about safety, real-life environments and proper error-handling. Could be optional.
function is_command () {
log "[I] check if commad $1 exists"
type "$1" &> /dev/null ;
}
CMD=zip
if is_command ${CMD} ; then
log "[I] '${CMD}' command found"
else
log "[E] '${CMD}' command not found"
fi
Template processing
Could be only my subjective opinion, but anyway. I used several different ways to generate some configuration/etc right from the script. Perl, sed and others do the job, but look a little bit scary.
Recently I noticed a better way:
function process_template() {
source $1 > $2
result=$?
if [ $result -ne 0 ]; then
log "[E] Error during template processing: '$1' > '$2'"
fi
return $result
}
VALUE1="tmpl-value-1"
VALUE2="tmpl-value-2"
VALUE3="tmpl-value-3"
process_template template.tmpl template.result
Template example
echo "Line1: ${VALUE1}
Line2: ${VALUE2}
Line3: ${VALUE3}"
Result example
Line1: tmpl-value-1
Line2: tmpl-value-2
Line3: tmpl-value-3

Answer 6

There is no more helpful thing to a shell script than a well documented behaviour with examples of usage and a list of known bugs. I think that no one program can be titled bulletproof, and bugs may appear in every moment (especially when your script gets used by the other people), so the only thing I’m taking care of is the good coding style and using only these things that the script really need. You’re standing on the way of aggregating, and it always going to become a large system which comes with a lot of unused modules, which is hard to port and hard to support. And the more system trying to be portable, the bigger it grows. Seriously, shell scripts do not need it to be implemented in that way. They must be kept as small as possible to simplify further use.
If the system really needs something big and bulletproof, it’s time to think about C99 or even C++.