In bash i'm building an update script, how to update the updater script - bash

I am building a little script to update application files on a raspberry pi.
It will do the following:
Download a zip file of the application files
Unzip them
Copy each one to the right place and make it executable etc as needed.
The problem i'm having is that one of the files is updatescript.sh.
I've read that it is dangerous to update / change a bash script while it is executing. See Edit shell script while it's running
Is there a good way to achieve what I'm trying to do?

What you've read is badly overblown.
It's completely safe to overwrite a shell script in-place by mving a different file over it. When you do this, the old file handle is still valid, referring to the original unmodified file contents. What you can't safely do is edit the existing file in-place.
So, the below is fine (and is what all your OS-vendor update tools like RPM do in effect):
#!/usr/bin/env bash
tempfile=$(mktemp "$BASH_SOURCE".XXXXXX)
if curl https://example.com/whatever >"$tempfile" &&
curl https://example.com/whatever.sig >"$tempfile.sig" &&
gpgv "$tempfile.sig" "$tempfile"; then
chown --reference="$BASH_SOURCE" -- "$tempfile"
chmod --reference="$BASH_SOURCE" -- "$tempfile"
sync # force your filesystem to fully flush file contents to disk
mv -- "$tempfile" "$BASH_SOURCE" && rm -f -- "$tempfile.sig"
else
rm -f -- "$tempfile" "$tempfile.sig"
exit 1
fi
...whereas this is risky:
curl https://example.com/whatever >/usr/local/bin/whatever
So do the first, thing, not the second one: When downloading a new version of your script, write that to a different file, and only rename it over the original when the download succeeded. That's what you want to do anyhow to ensure atomicity.
(There are also some demonstrations of code-signing validation practices above because, well, you need them when building an updater. You wouldn't be trying to distribute code via an automated download without verifying a signature, right? Because that's how one simple breakin to your web server results in every single one of your customers being 0wned. The above expects the public side of your code-signing keys to be in ~/.gnupg/trustedkeys.gpg, but you can put trustedkeys.gpg in any directory and point to it with the environment variable GNUPGHOME).
Even if you don't write your update code safely, the risk is still trivial to mitigate. If you move the body of your script into a function, such that it has to be completely read before any part of it can be executed, then there's no part of the file that isn't already read at the time when execution begins.
#!/usr/bin/env bash
main() {
echo "Logic all goes here"
}; { main "$#"; exit; }
Because { main "$#"; exit; } is part of a compound command, the parser reads the exit before it starts executing the main, so it's guaranteed that no further source-file content will be read after main exits, even if some future bash release didn't handle input line-by-line in the first place.

Basically do something along:
shouldbe="/tmp/$(basename "$0")"
if [ "$0" != "$shouldbe" ]; then
cp "$0" "$shouldbe"
exec env REALPATH="$0" "$shouldbe" "$#"
fi
Check if you are running from a temporary directory
If you are not, copy yourself and rerun from the temporary directory
You can even pass some variables/state along, by using environmental variables or arguments. Then you can update yourself by using simple cp, as the old path isn't sourced (or even opened) anymore.
cp "new_script_version.sh" "$REALPATH"
The script simply looks like this:
#!/bin/bash
# we need to be run from /tmp directory
shouldbe="/tmp/$(basename "$0")"
if [ "$0" != "$shouldbe" ]; then
cp "$0" "$shouldbe"
exec env REALPATH="$0" "$shouldbe" "$#"
fi
echo "Updatting...."
echo "downloading zip files"
echo "unziping zip files..."
echo "Copying each zip files etc."
cp directory"new_updatescript.sh "$REALPATH"
echo "Update succedded"
Live/test version available at tutorialspoint.
One would also implement some flock locking to the scripts just in case.

Related

How to make folders for individual files within a directory via bash script?

So I've got a movie collection that's dumped into a single folder (I know, bad practice in retrospect.) I want to organize things a bit so I can use Radarr to grab all the appropriate metadata, but I need all the individual files in their own folders. I created the script below to try and automate the process a bit, but I get the following error.
Script
#! /bin/bash
for f in /the/path/to/files/* ;
do
[[ -d $f ]] && continue
mkdir "${f%.*}"
mv "$f" "${f%.*}"
done
EDIT
So I've now run the script through Shellcheck.net per the suggestion of Benjamin W. It doesn't throw any errors according to the site, though I still get the same errors when I try running the command.
EDIT 2*
No errors now, but the script does nothing when executed.
Assignments are evaluated only once, and not whenever the variable being assigned to is used, which I think is what your script assumes.
You could use a loop like this:
for f in /path/to/all/the/movie/files/*; do
mkdir "${f%.*}"
mv "$f" "${f%.*}"
done
This uses parameter expansion instead of cut to get rid of the file extension.

Insert delay between lines of bash

I have a very simple renaming script I'm running in OSX Terminal. It looks like this:
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1140122_alternate1.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1140122_alternate1A.tif
I usually have several hundred lines of rename code like this one for all the files I have to rename.
However I think the network security at work is messing with the code because it will randomly jack up the file names. I think it's interrupting the code, the code is so simple I can't think of another reason why it wouldn't work.
I want to try adding a 1sec delay between each line, but how? I've read that something like sleep 1s might work but do I have to add that between every single line? That's going to be a headache if that's the case. If it is, is there another way?
UPDATE: I have a delay working but still getting the same problems as before. This is what Terminal returns:
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate1.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate1A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate2.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate2A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate3.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Remv -nvest/1247136_alternate3A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_alternate4.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Remv -nTest/1247136_alternate4A.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_lifestyle.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Renmv -nv /Volume36_lifestyleA.tif
mv -nv /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_standard.tif /Volumes/COMMON-LIC-PHOTO/DATA/James/Rename_Test/1247136_standardA.tifç^C^C^C^C^C
It's throwing up all kinds of junk in the rename part. It's messing with the file names and the directory names and I can't figure out why.
If you are planing to perform all those mv commands from terminal you can make a bash alias:
alias mvd='sleep 2s && mv'
In terms of a script, since scripts do not understand bash alias (at least easily) you can built a similar function in the beginning of your script:
function mvd { sleep 2s && mv "$#"; }
The only thing you need to do is to use the new mvd command instead of mv.
Tip: In case of alias you can also name your alias mv (same name as the command).
If you already have a script that has hardcoded paths (eg, the script looks like:
mv -nv /path1 /path2
mv -nv /path3 /path4
...
Then probably the simplest thing to do would be to define a function at the top of the script by adding:
mv() { command mv "$#"; sleep 1; }
the following script just reads in your file of commands and inserts a sleep after each command
while read curr_line; do
echo curr_line $curr_line
return_msg=$( $curr_line ) # execute cmd
# may want to do error checking on value of error variable $? and return_msg
sleep 1
done < ./input_file_of_original_cmds.txt # read in that file

Bash If then that reads a list in a file condition

Here is the condition:
I have a file with all packages installed.
I have a folder with all kinds of other packages, but they include all of the ones in the list, plus more.
I need a bash script that will read the file and check a folder for packages that don't exist in the list then remove them, they are not needed, but keep the packages that are on the list in that folder.
Or perhaps the bash should read folder then if packages in the folder aren't on the list them rm -f that or those packages.
I am familiar with writing if then conditional statements, I just don't know how to do if making the items in the list a variable or variables (in a loop).
thanks!
I would move the packages on the list to a new folder, delete the original folder, and move the temporary folder back:
DIR=directory-name
mkdir "$DIR-tmp"
while read pkgname; do
if [[ -f "$DIR/$pkgname" ]]; then
mv "$DIR/$pkgname" "$DIR-tmp"
fi
done < package-list.txt
# Confirm $DIR-tmp has the files you want first!
rm -rf "$DIR"
mv "$DIR-tmp" "$DIR"
I think you want something like this:
for file in $(ls folder) ; do
grep -E "$file" install-list-file >/dev/null || \
echo $file
done > rm-list
vi rm-list # view file to ensure correct
rm $(<rm_list)
There are ways to make this faster (using parameter substitution to avoid fork/exec's), but I recommend avoiding fancy shell stuff [${file##*/}] until you've got the basics down. Also, this script basically translates the description into a script and is not intended to be much more than a guide on how to approach the problem.

writing a shell script if statement to check for directory

I need to write a script that will recreate my opt folder if it gets deleted when I remove a package from it. Here's a link to my previous post: dpkg remove to stop processes
Now, the issue I'm running into could be better described here: http://lists.debian.org/debian-devel/2006/03/msg00242.html
I was thinking of just adding a postrem script which checks if an opt directory exists, and if not, creates one. My experience with shell scripts is pretty limited though..
[ -d "$dir" ] || mkdir -p "$dir"
This could be written more verbosely / clearly as:
if ! test -d "$dir"; then
mkdir -p "$dir"
fi
See help test for more information.

Is this a valid self-update approach for a bash script?

I'm working on a script that has gotten so complex I want to include an easy option to update it to the most recent version. This is my approach:
set -o errexit
SELF=$(basename $0)
UPDATE_BASE=http://something
runSelfUpdate() {
echo "Performing self-update..."
# Download new version
wget --quiet --output-document=$0.tmp $UPDATE_BASE/$SELF
# Copy over modes from old version
OCTAL_MODE=$(stat -c '%a' $0)
chmod $OCTAL_MODE $0.tmp
# Overwrite old file with new
mv $0.tmp $0
exit 0
}
The script seems to work as intended, but I'm wondering if there might be caveats with this kind of approach. I just have a hard time believing that a script can overwrite itself without any repercussions.
To be more clear, I'm wondering, if, maybe, bash would read and execute the script line-by-line and after the mv, the exit 0 could be something else from the new script. I think I remember Windows behaving like that with .bat files.
Update: My original snippet did not include set -o errexit. To my understanding, that should keep me safe from issues caused by wget.
Also, in this case, UPDATE_BASE points to a location under version control (to ease concerns).
Result: Based on the input from these answers, I constructed this revised approach:
runSelfUpdate() {
echo "Performing self-update..."
# Download new version
echo -n "Downloading latest version..."
if ! wget --quiet --output-document="$0.tmp" $UPDATE_BASE/$SELF ; then
echo "Failed: Error while trying to wget new version!"
echo "File requested: $UPDATE_BASE/$SELF"
exit 1
fi
echo "Done."
# Copy over modes from old version
OCTAL_MODE=$(stat -c '%a' $SELF)
if ! chmod $OCTAL_MODE "$0.tmp" ; then
echo "Failed: Error while trying to set mode on $0.tmp."
exit 1
fi
# Spawn update script
cat > updateScript.sh << EOF
#!/bin/bash
# Overwrite old file with new
if mv "$0.tmp" "$0"; then
echo "Done. Update complete."
rm \$0
else
echo "Failed!"
fi
EOF
echo -n "Inserting update process..."
exec /bin/bash updateScript.sh
}
(At least it doesn't try to continue running after updating itself!)
The thing that makes me nervous about your approach is that you're overwriting the current script (mv $0.tmp $0) as it's running. There are a number of reasons why this will probably work, but I wouldn't bet large amounts that it's guaranteed to work in all circumstances. I don't know of anything in POSIX or any other standard that specifies how the shell processes a file that it's executing as a script.
Here's what's probably going to happen:
You execute the script. The kernel sees the #!/bin/sh line (you didn't show it, but I presume it's there) and invokes /bin/sh with the name of your script as an argument. The shell then uses fopen(), or perhaps open() to open your script, reads from it, and starts interpreting its contents as shell commands.
For a sufficiently small script, the shell probably just reads the whole thing into memory, either explicitly or as part of the buffering done by normal file I/O. For a larger script, it might read it in chunks as it's executing. But either way, it probably only opens the file once, and keeps it open as long as it's executing.
If you remove or rename a file, the actual file is not necessarily immediately erased from disk. If there's another hard link to it, or if some process has it open, the file continues to exist, even though it may no longer be possible for another process to open it under the same name, or at all. The file is not physically deleted until the last link (directory entry) that refers to it has been removed, and no processes have it open. (Even then, its contents won't immediately be erased, but that's going beyond what's relevant here.)
And furthermore, the mv command that clobbers the script file is immediately followed by exit 0.
BUT it's at least conceivable that the shell could close the file and then re-open it by name. I can't think of any good reason for it to do so, but I know of no absolute guarantee that it won't.
And some systems tend to do stricter file locking that most Unix systems do. On Windows, for example, I suspect that the mv command would fail because a process (the shell) has the file open. Your script might fail on Cygwin. (I haven't tried it.)
So what makes me nervous is not so much the small possibility that it could fail, but the long and tenuous line of reasoning that seems to demonstrate that it will probably succeed, and the very real possibility that there's something else I haven't thought of.
My suggestion: write a second script whose one and only job is to update the first. Put the runSelfUpdate() function, or equivalent code, into that script. In your original script, use exec to invoke the update script, so that the original script is no longer running when you update it. If you want to avoid the hassle of maintaining, distributing, and installing two separate scripts. you could have the original script create the update script with a unique in /tmp; that would also solve the problem of updating the update script. (I wouldn't worry about cleaning up the autogenerated update script in /tmp; that would just reopen the same can of worms.)
Yes, but ... I would recommend you keep a more layered version of your script's history, unless the remote host can also perform version-control with histories. That being said, to respond directly to the code you have posted, see the following comments ;-)
What happens to your system when wget has a hiccup, quietly overwrites part of your working script with only a partial or otherwise corrupt copy? Your next step does a mv $0.tmp $0 so you've lost your working version. (I hope you have it in version control on the remote!)
You can check to see if wget returns any error messages
if ! wget --quiet --output-document=$0.tmp $UPDATE_BASE/$SELF ; then
echo "error on wget on $UPDATE_BASE/$SELF"
exit 1
fi
Also, Rule-of-thumb tests will help, i.e.
if (( $(wc -c < $0.tmp) >= $(wc -c < $0) )); then
mv $0.tmp $0
fi
but are hardly foolproof.
If your $0 could windup with spaces in it, better to surround all references like "$0".
To be super-bullet proof, consider checking all command returns AND that Octal_Mode has a reasonable value
OCTAL_MODE=$(stat -c '%a' $0)
case ${OCTAL_MODE:--1} in
-[1] )
printf "Error : OCTAL_MODE was empty\n"
exit 1
;;
777|775|755 ) : nothing ;;
* )
printf "Error in OCTAL_MODEs, found value=${OCTAL_MODE}\n"
exit 1
;;
esac
if ! chmod $OCTAL_MODE $0.tmp ; then
echo "error on chmod $OCTAL_MODE %0.tmp from $UPDATE_BASE/$SELF, can't continue"
exit 1
fi
I hope this helps.
Very late answer here, but as I just solved this too, I thought it might help someone to post the approach:
#!/usr/bin/env bash
#
set -fb
readonly THISDIR=$(cd "$(dirname "$0")" ; pwd)
readonly MY_NAME=$(basename "$0")
readonly FILE_TO_FETCH_URL="https://your_url_to_downloadable_file_here"
readonly EXISTING_SHELL_SCRIPT="${THISDIR}/somescript.sh"
readonly EXECUTABLE_SHELL_SCRIPT="${THISDIR}/.somescript.sh"
function get_remote_file() {
readonly REQUEST_URL=$1
readonly OUTPUT_FILENAME=$2
readonly TEMP_FILE="${THISDIR}/tmp.file"
if [ -n "$(which wget)" ]; then
$(wget -O "${TEMP_FILE}" "$REQUEST_URL" 2>&1)
if [[ $? -eq 0 ]]; then
mv "${TEMP_FILE}" "${OUTPUT_FILENAME}"
chmod 755 "${OUTPUT_FILENAME}"
else
return 1
fi
fi
}
function clean_up() {
# clean up code (if required) that has to execute every time here
}
function self_clean_up() {
rm -f "${EXECUTABLE_SHELL_SCRIPT}"
}
function update_self_and_invoke() {
get_remote_file "${FILE_TO_FETCH_URL}" "${EXECUTABLE_SHELL_SCRIPT}"
if [ $? -ne 0 ]; then
cp "${EXISTING_SHELL_SCRIPT}" "${EXECUTABLE_SHELL_SCRIPT}"
fi
exec "${EXECUTABLE_SHELL_SCRIPT}" "$#"
}
function main() {
cp "${EXECUTABLE_SHELL_SCRIPT}" "${EXISTING_SHELL_SCRIPT}"
# your code here
}
if [[ $MY_NAME = \.* ]]; then
# invoke real main program
trap "clean_up; self_clean_up" EXIT
main "$#"
else
# update myself and invoke updated version
trap clean_up EXIT
update_self_and_invoke "$#"
fi

Resources