I am reading makefile tutorials and I don't see how I can run unix or file system commands. This is my makefile which works right now:
push:
docker build -t dataengineering/dataloader .
docker tag dataengineering/dataloader:latest 11111111111.dkr.ecr.us-west-2.amazonaws.com/dataengineering/dataloader:latest
docker push 1111111111.dkr.ecr.us-west-2.amazonaws.com/dataengineering/dataloader:latest
deploy:
#if [ ! "$(environment)" ]; then echo "environment must be defined" && exit 1; fi
#if [ ! "$(target)" ]; then echo "target must be defined" && exit 1; fi
kubectl delete deploy dataloader-$(target) -n dataengineering|| continue
kubectl apply -f kube/$(environment)/deployment-$(target).yaml -n dataengineering
and I want to add cp -r ../datastore/ . to the beginning of push like this:
push:
cp -r ../datastore/ .
docker build -t dataengineering/dataloader .
docker tag dataengineering/dataloader:latest 11111111111.dkr.ecr.us-west-2.amazonaws.com/dataengineering/dataloader:latest
docker push 1111111111.dkr.ecr.us-west-2.amazonaws.com/dataengineering/dataloader:latest
Can I do this? Where does it say I can write unix/file commads in a makefile?
Yes you can. Not sure if there is any specification on limitations, but the GNU Makefile conventions mention file operation utilities throughout the docs which might be a sufficient hint that it's a feasible approach.
I.e. from the section Utilities in Makefiles
File management utilities such as ln, rm, mv, and so on, need not be referred to through variables in this way, since users don’t need to replace them with other programs.
Related
My Docker wrapper script works as intended when the current working directory does not contain spaces, however there is a bug when it does.
I have simplified an example to make use of the smallest official Docker image I could find and a well known GNU core utility. Of course this example is not very useful. In my real world use case, a much more complicated environment is packaged.
Docker Wrapper Script:
#!/usr/bin/env bash
##
## Dockerized ls
##
set -eux
# Only allocate tty if one is detected
# See https://stackoverflow.com/questions/911168/how-to-detect-if-my-shell-script-is-running-through-a-pipe
if [[ -t 0 ]]; then
DOCKER_RUN_OPTIONS+="-i "
fi
if [[ -t 1 ]]; then
DOCKER_RUN_OPTIONS+="-t "
fi
WORK_DIR="$(realpath .)"
DOCKER_RUN_OPTIONS+="--rm --user=$(id -u $(logname)):$(id -g $(logname)) --workdir=${WORK_DIR} --mount type=bind,source=${WORK_DIR},target=${WORK_DIR}"
exec docker run ${DOCKER_RUN_OPTIONS} busybox:latest ls "$#"
You can save this somewhere as /tmp/docker_ls for example. Remember to chmod +x /tmp/docker_ls
Now you are able to use this Dockerized ls in any path which contains no spaces as follows:
/tmp/docker_ls -lah
/tmp/docker_ls -lah | grep 'r'
Note that /tmp/docker_ls -lah /path/to/something is not implemented. The wrapper script would have to be adapted to parse parameters and mount the path argument into the container.
Can you see why this would not work when current working directory path contains spaces? What can be done to rectify it?
Solution:
#david-maze's answer solved the problem. Please see: https://stackoverflow.com/a/55763212/1782641
Using his advice I refactored my script as follows:
#!/usr/bin/env bash
##
## Dockerized ls
##
set -eux
# Only allocate tty if one is detected. See - https://stackoverflow.com/questions/911168
if [[ -t 0 ]]; then IT+=(-i); fi
if [[ -t 1 ]]; then IT+=(-t); fi
USER="$(id -u $(logname)):$(id -g $(logname))"
WORKDIR="$(realpath .)"
MOUNT="type=bind,source=${WORKDIR},target=${WORKDIR}"
exec docker run --rm "${IT[#]}" --user "${USER}" --workdir "${WORKDIR}" --mount "${MOUNT}" busybox:latest ls "$#"
If your goal is to run a process on the current host directory as the current host user, you will find it vastly easier and safer to use a host process, and not an isolation layer like Docker that intentionally tries to hide these things from you. For what you’re showing I would just skip Docker and run
#!/bin/sh
ls "$#"
Most software is fairly straightforward to install without Docker, either using a package manager like APT or filesystem-level isolation like Python’s virtual environments and Node’s node_modules directory. If you’re writing this script then Docker is just getting in your way.
In a portable shell script there’s no way to make “a list of words” in a way that keeps their individual wordiness. If you know you’ll always want to pass some troublesome options then this is still fairly straightforward: include them directly in the docker run command and don’t try to create a variable of options.
#!/bin/sh
RM_IT="--rm"
if [[ -t 0 ]]; then RM_IT="$RM_IT -i"; fi
if [[ -t 1 ]]; then RM_IT="$RM_IT -t"; fi
UID=$(id -u $(logname))
GID=$(id -g $(logname))
# We want the --rm -it options to be expanded into separate
# words; we want the volume options to stay as a single word
docker run $RM_IT "-u$UID:$GID" "-w$PWD" "-v$PWD:$PWD" \
busybox \
ls "$#"
Some shells like ksh, bash, and zsh have array types, but these shells may not be present on every system or environment (your busybox image doesn’t have any of these for example). You also might consider picking a higher-level scripting language that can more explicitly pass words into an exec type call.
I'm taking a stab at this to give you something to try:
Change this:
DOCKER_RUN_OPTIONS+="--rm --user=$(id -u $(logname)):$(id -g $(logname)) --workdir=${WORK_DIR} --mount type=bind,source=${WORK_DIR},target=${WORK_DIR}"
To this:
DOCKER_RUN_OPTIONS+="--rm --user=$(id -u $(logname)):$(id -g $(logname)) --workdir=${WORK_DIR} --mount type=bind,source='${WORK_DIR}',target='${WORK_DIR}'"
Essentially, we are putting the ' in there to escape the space when the $DOCKER_RUN_OPTIONS variable is evaluated by bash on the 'exec docker' command.
I haven't tried this - it's just a hunch / first shot.
I am building a little script to update application files on a raspberry pi.
It will do the following:
Download a zip file of the application files
Unzip them
Copy each one to the right place and make it executable etc as needed.
The problem i'm having is that one of the files is updatescript.sh.
I've read that it is dangerous to update / change a bash script while it is executing. See Edit shell script while it's running
Is there a good way to achieve what I'm trying to do?
What you've read is badly overblown.
It's completely safe to overwrite a shell script in-place by mving a different file over it. When you do this, the old file handle is still valid, referring to the original unmodified file contents. What you can't safely do is edit the existing file in-place.
So, the below is fine (and is what all your OS-vendor update tools like RPM do in effect):
#!/usr/bin/env bash
tempfile=$(mktemp "$BASH_SOURCE".XXXXXX)
if curl https://example.com/whatever >"$tempfile" &&
curl https://example.com/whatever.sig >"$tempfile.sig" &&
gpgv "$tempfile.sig" "$tempfile"; then
chown --reference="$BASH_SOURCE" -- "$tempfile"
chmod --reference="$BASH_SOURCE" -- "$tempfile"
sync # force your filesystem to fully flush file contents to disk
mv -- "$tempfile" "$BASH_SOURCE" && rm -f -- "$tempfile.sig"
else
rm -f -- "$tempfile" "$tempfile.sig"
exit 1
fi
...whereas this is risky:
curl https://example.com/whatever >/usr/local/bin/whatever
So do the first, thing, not the second one: When downloading a new version of your script, write that to a different file, and only rename it over the original when the download succeeded. That's what you want to do anyhow to ensure atomicity.
(There are also some demonstrations of code-signing validation practices above because, well, you need them when building an updater. You wouldn't be trying to distribute code via an automated download without verifying a signature, right? Because that's how one simple breakin to your web server results in every single one of your customers being 0wned. The above expects the public side of your code-signing keys to be in ~/.gnupg/trustedkeys.gpg, but you can put trustedkeys.gpg in any directory and point to it with the environment variable GNUPGHOME).
Even if you don't write your update code safely, the risk is still trivial to mitigate. If you move the body of your script into a function, such that it has to be completely read before any part of it can be executed, then there's no part of the file that isn't already read at the time when execution begins.
#!/usr/bin/env bash
main() {
echo "Logic all goes here"
}; { main "$#"; exit; }
Because { main "$#"; exit; } is part of a compound command, the parser reads the exit before it starts executing the main, so it's guaranteed that no further source-file content will be read after main exits, even if some future bash release didn't handle input line-by-line in the first place.
Basically do something along:
shouldbe="/tmp/$(basename "$0")"
if [ "$0" != "$shouldbe" ]; then
cp "$0" "$shouldbe"
exec env REALPATH="$0" "$shouldbe" "$#"
fi
Check if you are running from a temporary directory
If you are not, copy yourself and rerun from the temporary directory
You can even pass some variables/state along, by using environmental variables or arguments. Then you can update yourself by using simple cp, as the old path isn't sourced (or even opened) anymore.
cp "new_script_version.sh" "$REALPATH"
The script simply looks like this:
#!/bin/bash
# we need to be run from /tmp directory
shouldbe="/tmp/$(basename "$0")"
if [ "$0" != "$shouldbe" ]; then
cp "$0" "$shouldbe"
exec env REALPATH="$0" "$shouldbe" "$#"
fi
echo "Updatting...."
echo "downloading zip files"
echo "unziping zip files..."
echo "Copying each zip files etc."
cp directory"new_updatescript.sh "$REALPATH"
echo "Update succedded"
Live/test version available at tutorialspoint.
One would also implement some flock locking to the scripts just in case.
I was looking at how to use runit to run gunicorn. I was looking at the bash file and I don't know what -f $PID does in
#!/bin/sh
GUNICORN=/usr/local/bin/gunicorn
ROOT=/path/to/project
PID=/var/run/gunicorn.pid
APP=main:application
if [ -f $PID ]; then rm $PID; fi
cd $ROOT
exec $GUNICORN -c $ROOT/gunicorn.conf.py --pid=$PID $APP
Google is useless in this case because searching for flags is useless
Google is useless in this case because searching for flags is useless
Fortunately, the Bash Reference Manual is available online, at http://www.gnu.org/software/bash/manual/bashref.html. It's the first hit when you Google for "Bash manual". §6.4 "Bash Conditional Expressions" says:
-f file
True if file exists and is a regular file.
-f - file is a regular file (not a directory or device file)
Check this out for all file test operators:
http://tldp.org/LDP/abs/html/fto.html
The [ is the same as the command test which allows you to test certain things. Try help test to find out what the flags are. Things to be careful with are spaces - the [ needs a space after it.
-f checks if the file exists and is a regular file.
[ -f "$var" ]
Checks if $var is an existing file (regular file). Symbolic link passes this test too.
I need to write a script that will recreate my opt folder if it gets deleted when I remove a package from it. Here's a link to my previous post: dpkg remove to stop processes
Now, the issue I'm running into could be better described here: http://lists.debian.org/debian-devel/2006/03/msg00242.html
I was thinking of just adding a postrem script which checks if an opt directory exists, and if not, creates one. My experience with shell scripts is pretty limited though..
[ -d "$dir" ] || mkdir -p "$dir"
This could be written more verbosely / clearly as:
if ! test -d "$dir"; then
mkdir -p "$dir"
fi
See help test for more information.
I'm trying to store all my profile configuration files (~/.xxx) in git. I'm pretty horrible at bash scripting but I imagine this will be pretty straight forward for you scripting gurus.
Basically, I'd like a script that will create symbolic links in my home directory to files in my repo. Twist is, I'd like it warn and prompt for overwrite if the symlink will be overwriting an actual file. It should also prompt if a sym link is going to be overwritten, but the target path is different.
I don't mind manually editing the script for each link I want to create. I'm more concerned with being able to quickly deploy new config scripts by running this script stored in my repo.
Any ideas?
The ln command is already conservative about erasing, so maybe the KISS approach is good enough for you:
ln -s git-stuff/home/.[!.]* .
If a file or link already exists, you'll get an error message and this link will be skipped.
If you want the files to have a different name in your repository, pass the -n option to ln so that it doesn't accidentally create a symlink in an existing subdirectory of that name:
ln -sn git-stuff/home/profile .profile
...
If you also want to have links in subdirectories of your home directory, cp -as reproduces the directory structure but creates symbolic links for regular files. With the -i option, it prompts if a target already exists.
cp -i -as git-stuff/home/.[!.]* .
(My answer assumes GNU ln and GNU cp, such as you'd find on Linux (and Cygwin) but usually not on other unices.)
The following has race conditions, but it is probably as safe as you can get without filesystem transactions:
# create a symlink at $dest pointing to $source
# not well tested
set -e # abort on errors
if [[ ( -h $dest && $(readlink -n "$dest") != $source ) || -f $dest || -d $dest ]]
then
read -p "Overwrite $dest? " answer
else
answer=y
fi
[[ $answer == y ]] && ln -s -n -f -v -- "$source" "$dest"