Testing server setup bash scripts - bash

I'm just learning to write bash scripts.
I'm writing a script to setup a new server.
How should I go about testing the script.
i.e.
I use apt install for certain packages like apache, php etc. and then a couple of lines down there is an error.
I then need to fix the error and run it again but it will run all the install commands again.
The system will probably say the package is installed already, but what if there are commands which append strings to files.
If these are run again it will append the same string to the file a second time.
What is the best approach to write bash-scripts like this?
Can you do test runs which rollback everything after an error or end of the script?
Or even better to have the script continue from the line where the error occured the next time it is run?
I'm doing this on an Ubuntu 18.04 server.

it's a matter of how clear you want it to be to read it, but
[ -f .step01-done ] || your install command && touch .step01-done
[ -f .step02-done ] || your other install command && touch .step02-done
maybe a little easier to read:
if ! [ -f .step01-done ]; then
if your install command ; then
touch .step01-done
fi
fi
if ! [ -f .step02-done ]; then
if your other install command ; then
touch .step02-done
fi
fi
...or something in between.
Now, I would suggest creating a directory somewhere and maybe logging output from the commands to some file there (maybe tee it) but definitely putting all these files you are creating with touch there. That way if you start it from another directory by accident, it won't matter. You just need to make sure that apt-get or whatever you use actual returns false if it fails. It should.
You could even make a function that does it in a nice way...
#!/bin/bash
function do_cmd() {
if [ -f "$1.done" ]; then
echo "$2: skipping already completed step"
return 0
fi
echo -n "$2: "
$3 1> "$1.out" 2> "$1.err"
if $?; then
echo "ok"
touch "$1.done"
return 0
else
echo "failed"
echo -e "see \"$1.out\" and/or \"$1.err\" for details."
return 1
# could "exit 1" instead
fi
}
[ -d /root/mysetup ] || mkdir /root/mysetup
if ! [ -d /root/mysetup ]; then
echo "failed to find or create /root/mysetup directory
exit 1
fi
cd /root/mysetup
# ---------------- your steps go here -------------------
do_cmd prog1 "installing prog1" "apt-get install prog1" || exit 1
do_cmd prog2 "installing prog2" "apt-get install prog2" || exit 1
do_cmd startfoo "starting foo service" "service foo start" || exit 1
echo "all setup functions finished."
You would use:
do_cmd identifier "description" "command or function"
description
identifier: unique identifier used when files are generated:
identifier.out: standard output from command
identifier.err: standard error from command
identifier.done: created when command is successful
description: this is actually printed to the terminal when the step is being executed.
command or function: this is the actual command to run
not sure why stackoverflow forced me to format that last bit as code but w/e

Related

Understanding Bash if statement that invokes a command

Does anyone know what this is doing:
if ! /fgallery/fgallery -v -j3 /images /usr/share/nginx/html/ "${GALLERY_TITLE:-Gallery}"; then
mkdir -p /usr/share/nginx/html
I understand the first part is saying if /fgallery/fgallery directory doesn't exist but after this it it not clear for me.
In Bash, we can build an if based on the exit status of a command this way:
if command; then
echo "Command succeeded"
else
echo "Command failed"
fi
then part is executed when the command exits with 0 and else part otherwise.
Your code is doing exactly that.
It can be rewritten as:
/fgallery/fgallery -v -j3 /images /usr/share/nginx/html/ "${GALLERY_TITLE:-Gallery}"; fgallery_status=$?
if [ "$fgallery_status" -ne 0 ]; then
mkdir -p /usr/share/nginx/html
fi
But the former construct is more elegant and less error prone.
See these posts:
How to conditionally do something if a command succeeded or failed
Why is testing "$?" to see if a command succeeded or not, an antipattern?

Bash - A way to watch files for changes?

My current solution is to use the python library watchdog and the bash snippet (originally taken from here).
watchmedo shell-command client/js/src/templates/ proto/ --recursive \
--patterns="*.soy;*.proto" \
--command="echo \"WATCHMEDO file changed - rebuilding\"; make genfiles;"
Basically I'm watching a few template files, and then running make genfiles automatically if one of them changes.
I'm wondering if there's a way to do this in pure bash? I'd rather not have all my devs have to depend on that Python library.
I'm on OSX.
#!/bin/bash
watched_files=$# # pass watched files as cmd line arguments
if [ -z "$watched_files" ]; then
echo "Nothing to watch, abort"
exit
else
echo "watching: $watched_files"
fi
previous_checksum="dummy"
while [ 1 ]; do
checksum=$(md5 $watched_files | md5) # I use Mac so I have `md5`, in linux it's `md5sum`
if [ "$checksum" != "$previous_checksum" ]; then
echo "None shall pass!" # do your stuff here
fi
previous_checksum="$checksum"
sleep 1
done
This is a nice cli FAM client: http://fileschanged.sourceforge.net/

Run wget and other commands in shell script

I'm trying to create a shell script that I will download the latest Atomic gotroot rules to my server, unpack them, copy them to the correct folder, etc.,
I've been reading shell tutorials and forum posts for most of the day and the syntax escapes me for some of these. I have run all these commands and I know they work if I manually run them.
I know I need to develop some error checking, but I'm just trying to get the commands to run correctly. The main problem at the moment is the syntax of the wget commands, i've got errors about missing semi-colons, divide by zero, unsupported schemes - I've tried various quoting (single and double) and escaping - / " characters in various combinations.
Thanks for any help.
The raw wget command is
wget --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/VERSION"
#!/bin/sh
update_modsec_rules(){
wget=/usr/bin/wget
tar=/bin/tar
apachectl=/usr/bin/apache2ctl
TXT="Script Run Finished"
WORKING_DIR="/var/asl/updates"
TARGET_DIR="/usr/local/apache/conf/modsec_rules/"
EXISTING_FILES="/var/asl/updates/modsec/*"
EXISTING_ARCH="/var/asl/updates/modsec-*"
WGET_OPTS='--user=jim --password=xxx-yyy-zzz'
URL_BASE="http://updates.atomicorp.com/channels/rules/subscription"
# change to working directory and cleanup any downloaded files and extracted rules in modsec/ directory
cd $WORKING_DIR
rm -f $EXISTING_ARCH
rm -f $EXISTING_FILES
rm -f VERSION*
# wget to download VERSION file
$wget ${WGET_OPTS} "${URL_BASE}/VERSION"
# get current MODSEC_VERSION from VERSION file and save as variable
source VERSION
TARGET_DATE=$MODSEC_VERSION
echo $TARGET_DATE
# wget to download current archive
$wget ${WGET_OPTS} "${URL_BASE}/modsec-${TARGET_DATE}.tar.gz"
# extract archive
echo "extracting files . . . "
tar zxvf $WORKING_DIR/modsec-${TARGET_DATE}.tar.gz
echo "copying files . . . "
cp -uv $EXISTING_FILES $TARGET_DIR
echo $TXT
}
update_modsec_rules $# 2>&1 | tee -a /var/asl/modsec_update.log
RESTART_APACHE="/usr/local/cpanel/scripts/restartsrv httpd"
$RESTART_APACHE
Here are some guidelines to use when writing shell scripts.
Always quote variables when you use them. This helps avoid the possibility of misinterpretation. (What if a filename contains a space?)
Don't trust fileglobbing on commands like rm. Use for loops instead. (What if a filename starts with a hyphen?)
Avoid subshells when possible. Your lines with backquotes make me itchy.
Don't exec if you can help it. And especially don't expect any parts of your script after your exec to actually get run.
I should point out that while your shell may be bash, you've specified /bin/sh for execution of this script, so it is NOT a bash script.
Here's a rewrite with some error checking. Add salt to taste.
#!/bin/sh
# Linux
wget=/usr/bin/wget
tar=/bin/tar
apachectl=/usr/sbin/apache2ctl
# FreeBSD
#wget=/usr/local/bin/wget
#tar=/usr/bin/tar
#apachectl=/usr/local/sbin/apachectl
TXT="GOT TO THE END, YEAH"
WORKING_DIR="/var/asl/updates"
TARGET_DIR="/usr/local/apache/conf/modsec_rules/"
EXISTING_FILES_DIR="/var/asl/updates/modsec/"
EXISTING_ARCH="/var/asl/updates/"
URL_BASE="http://updates.atomicorp.com/channels/rules/subscription"
WGET_OPTS='--user="jim" --password="xxx-yyy-zzz"'
if [ ! -x "$wget" ]; then
echo "ERROR: No wget." >&2
exit 1
elif [ ! -x "$apachectl" ]; then
echo "ERROR: No apachectl." >&2
exit 1
elif [ ! -x "$tar" ]; then
echo "ERROR: Not in Kansas anymore, Toto." >&2
exit 1
fi
# change to working directory and cleanup any downloaded files
# and extracted rules in modsec/ directory
if ! cd "$WORKING_DIR"; then
echo "ERROR: can't access working directory ($WORKING_DIR)" >&2
exit 1
fi
# Delete each file in a loop.
for file in "$EXISTING_FILES_DIR"/* "$EXISTING_ARCH_DIR"/modsec-*; do
rm -f "$file"
done
# Move old VERSION out of the way.
mv VERSION VERSION-$$
# wget1 to download VERSION file (replaces WGET1)
if ! $wget $WGET_OPTS $URL_BASE}/VERSION; then
echo "ERROR: can't get VERSION" >&2
mv VERSION-$$ VERSION
exit 1
fi
# get current MODSEC_VERSION from VERSION file and save as variable,
# but DON'T blindly trust and run scripts from an external source.
if grep -q '^MODSEC_VERSION=' VERSION; then
TARGET_DATE="`sed -ne '/^MODSEC_VERSION=/{s/^[^=]*=//p;q;}' VERSION`"
echo "Target date: $TARGET_DATE"
fi
# Download current archive (replaces WGET2)
if ! $wget ${WGET_OPTS} "${URL_BASE}/modsec-$TARGET_DATE.tar.gz"; then
echo "ERROR: can't get archive" >&2
mv VERSION-$$ VERSION # Do this, don't do this, I don't know your needs.
exit 1
fi
# extract archive
if [ ! -f "$WORKING_DIR/modsec-${TARGET_DATE}.tar.gz" ]; then
echo "ERROR: I'm confused, where's my archive?" >&2
mv VERSION-$$ VERSION # Do this, don't do this, I don't know your needs.
exit 1
fi
tar zxvf "$WORKING_DIR/modsec-${TARGET_DATE}.tar.gz"
for file in "$EXISTING_FILES_DIR"/*; do
cp "$file" "$TARGET_DIR/"
done
# So far so good, so let's restart apache.
if $apachectl configtest; then
if $apachectl restart; then
# Success!
rm -f VERSION-$$
echo "$TXT"
else
echo "ERROR: PANIC! Apache didn't restart. Notify the authorities!" >&2
exit 3
fi
else
echo "ERROR: Apache configs are broken. We're still running, but you'd better fix this ASAP." >&2
exit 2
fi
Note that while I've rewritten this to be more sensible, there is certainly still a lot of room for improvement.
You have two options:
1- changing this to
WGET1=' --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/VERSION"'
then run
wget $WGET1 same to WGET2
Or
2- encapsulating $WGET1 with backquotes ``.
e.g.:
`$WGET`
This applies to any command your executing out of a variable.
Suggested changes:
#!/bin/sh
TXT="GOT TO THE END, YEAH"
WORKING_DIR="/var/asl/updates"
TARGET_DIR="/usr/local/apache/conf/modsec_rules/"
EXISTING_FILES="/var/asl/updates/modsec/*"
EXISTING_ARCH="/var/asl/updates/modsec-*"
WGET1='wget --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/VERSION"'
WGET2='wget --user="jim" --password="xxx-yyy-zzz" "http://updates.atomicorp.com/channels/rules/subscription/modsec-$TARGET_DATE.tar.gz"'
## change to working directory and cleanup any downloaded files and extracted rules in modsec/ directory
cd $WORKING_DIR
rm -f $EXISTING_ARCH
rm -f $EXISTING_FILES
## wget1 to download VERSION file
`$WGET1`
## get current MODSEC_VERSION from VERSION file and save as variable
source VERSION
TARGET_DATE=`echo $MODSEC_VERSION`
## WGET2 command to download current archive
`$WGET2`
## extract archive
tar zxvf $WORKING_DIR/modsec-$TARGET_DATE.tar.gz
cp $EXISTING_FILES $TARGET_DIR
## restart server
exec '/usr/local/cpanel/scripts/restartsrv_httpd' $*;
Pro Tip: If you need string substitution, using ${VAR} is much better to eliminate ambiguity, e.g.:
tar zxvf $WORKING_DIR/modsec-${TARGET_DATE}.tar.gz

Prompt for `sudo` only if Bash script runs into "Permission denied"

Let's say I have a very simple script which creates a link in a certain directory, and kills the script if it fails.
ln -s "/opt/myapp" "${1}/link" || exit 1;
Right now it just quits if it runs into errors. I want to change it so only if it runs into permission errors when creating the link, it will execute the following lines instead of exiting:
echo "The target directory requires root privileges to access."
sudo ln -s "/opt/myapp" "${1}/myapp" || exit 1;
I don't want to prompt the users to run as root unless they absolutely have to.
ln seems to retun exit code 1 on failure regardless of whether it was a problem with permissions or any other errors such as a directory not existing, so I can't use that to detect which problem it ran into.
And if I instead store search through the output of ln for the string "Permission denied", I'm assuming it will fail on non-english operating systems.
I don't know of any ways to categorize ln exit reasons, or at least any documentation about specific exit codes you could test with $?, but you can test for relevant permissions with the standard test or [ command:
SOURCEFILE="/opt/myapp"
DESTDIR="${1}"
DESTTARGET="${DESTDIR}/myapp"
if [ ! -d "$DESTDIR" -o ! -e "$SOURCEFILE" ]; then
echo "Source file does not exist or destination directory does not exist." >&2
elif [ ! -r "$SOURCEFILE" -o ! -w "$DESTDIR" ]; then
echo "Source file is not readable or destination directory is not writable." >&2
# Run sudo command here
else
# Should work, run command here
fi

Bash: How to test for failure of mkdir command?

I'm writing a bash script and want to do robust error checking in it.
The exit status code for mv to make it fail is easy to simulate a failure. All you have to do is move a file that doesn't exist, and it fails.
However with mkdir I want to simulate it failing. mkdir could fail for any number of reasons, problems with the disk, or lack of permissions, but not sure how to simulate a failure.
Just use
mkdir your_directory/
if [ $? -ne 0 ] ; then
echo "fatal"
else
echo "success"
fi
where $? stands for the exit code from the last command executed.
To create parent directories, when these don't exist, run mkdir -p parent_directory/your_directory/
if ! mkdir your_directory 2>/dev/null; then
print_error
exit
fi
or
mkdir your_directory 2>/dev/null || { print_error; exit; }
mkdir will fail if the directory already exists (unless you are using -p), and return an error code of 1 (on my system), so create the directory first to test this on your own system. (Although I would assume that is standard across all shells.)
Alternatively, make the parent directory read-only.
in your script , you could also put a check for the new dir ....
mkdir -p new_dir ;
if [ -d new_dir ]
cd new_dir && ...... anything else you want .
else
echo "error in directory creation ";
exit 2 ;
fi
If you are lazy a simple set -e in the beginning of you script is enough. Often you just want to print an error and then terminate if something goes wrong.
Not exactly what you asked for, but perhaps what you want.

Resources