Shell script that watches HTML templates and compiles with Handlebars - bash

I'm trying to create a script that watches my HTML template files in a directory and when it notify's changes, it compiles the template. I can't get it working, this is what I got:
#!/bin/sh
while FILENAME=$(inotifywait --format %w -r templates/*.html)
do
COMPILED_FILE=$(echo "$FILENAME" | sed s/templates/templates\\/compiled/g | sed s/.html/.js/g)
handlebars $FILENAME -f $COMPILED_FILE -a
done
I use inotifywait to watch the current dir, although I want it also to check for sub directories. The compiled files then need to be saved in a sub directory called templates/compiled with optionally the sub directory.
So templates/foo.html needs to be compiled and stored as templates/compiled/foo.js
So templates/other/foo.html needs to be compiled and stored as templates/compiled/other/foo.js
As you can see I tried to watch the directoy and replace the templates/ name with templates/compiled.
Any help is welcome!

A few observations, then a solution:
Passing the argument -r templates/*.html only matches .html files in templates/ — not in templates/other/. Instead we're going to do -r templates which notifies us of changes to any file anywhere under templates.
If you don't use inotifywait in --monitor mode, you will miss any files that are changed in the brief period that handlebars is running (which could happen if you save all your open files at once). Better to do something like this:
#!/bin/bash
watched_dir="templates"
while read -r dirname events filename; do
printf 'File modified: %s\n' "$dirname$filename"
done < <(inotifywait --monitor --event CLOSE_WRITE --recursive "$watched_dir")
Then, as for transforming the paths, you could do something like:
$ dirname=templates/other/
$ echo "${dirname#*/}"
other/
$ echo "$watched_dir/compiled/${dirname#*/}"
templates/compiled/other/
$ filename=foo.html
$ echo "${filename%.html}"
foo
$ echo "${filename%.html}.js"
foo.js
$ echo "$watched_dir/compiled/${dirname#*/}${filename%.html}.js"
templates/compiled/other/foo.js
Notice that we can leverage Bash's builtin parameter expansion — no need for sed.
Putting it all together, we get:
#!/bin/bash
watched_dir="templates"
while read -r dirname events filename; do
[[ "${filename##*.}" != 'html' ]] && continue
output_path="$watched_dir/compiled/${dirname#*/}${filename%.html}.js"
handlebars "$dirname$filename" -f "$output_path" -a
done < <(inotifywait --monitor --event CLOSE_WRITE --recursive "$watched_dir")

Related

Trouble with cp command for directories

I'm trying to use the cp function to do copy directories:
src/1/b
src/2/d
src/3/c
src/4/a
src/5/e
then the copying should result in
tgt/a/4
tgt/b/1
tgt/c/3
tgt/d/2
tgt/e/5
I tried to use the 'basename' function as well as 'cp dir1/*dir2'. With the basename, do I make a loop to find every directory or is there a recursive builtin? Also tried the 'cp-r' recursive copy function. But nothing so far has worked.
I used tmp folder that will hols the SOURCE list of files, yo can readjust:
cat tmp
result:
src/1/b
src/2/d
src/3/c
src/4/a
src/5/e
from here, I echo out the command, but you can remove echo and it will execute, if this output seems correct:
#!/bin/bash
cat tmp |while read z
do
echo cp "$z" "tgt/$(echo "$z"|cut -d/ -f 3)/$(echo "$z"|cut -d/ -f 2)"
done
result:
cp src/1/b tgt/b/1
cp src/2/d tgt/d/2
cp src/3/c tgt/c/3
cp src/4/a tgt/a/4
cp src/5/e tgt/e/5
you can also add parameters to cp as you see fit. But first test with the echo command, then execute :)

inotifywait triggering event twice while converting docx to PDF

I have shell script with inotifwait set up as under:
inotifywait -r -e close_write,moved_to -m "<path>/upload" --format '%f######%e######%w'
There are some docx files residing in watched directory and some script converts docx to PDF via below command:
soffice --headless --convert-to pdf:writer_pdf_Export <path>/upload/somedoc.docx --outdir <path>/upload/
Somehow event is triggered twice as soon as PDF is generated. Entries are as under:
somedoc.pdf######CLOSE_WRITE,CLOSE######<path>/upload/
somedoc.pdf######CLOSE_WRITE,CLOSE######<path>/upload/
What else is wrong here?
Regards
It's triggered twice because this is how soffice appears to behave internally.
One day it may start writing it 10 times and doing sleep 2 between such writes during a single run, our program can't and I believe shouldn't anticipate it and depend on it.
So I'd try solving the problem from a different angle - lets just put the converted file into a temporary directory and then move it to the target dir, like this:
soffice --headless --convert-to pdf:writer_pdf_Export <path>/upload/somedoc.docx --outdir <path>/tempdir/ && mv <path>/tempdir/somedoc.pdf <path>/upload/
and use inotifywait in the following way:
inotifywait -r -e moved_to -m "<path>/upload" --format '%f######%e######%w'
The advantage is that you no longer depend on soffice's internal logic.
If you can't adjust behavior of the script producing the pdf files then indeed you'll need to resort to a workaround like #Tarun suggested.
I don't think you can control the external program as such. But I assume you are using this output for a pipe and then inputing it some place else. In that case you can avoid a event that happens continuously with a span of few seconds
So we add %T to --format and --timefmt "%s" to get the epoch time. Below is the updated command
$ inotifywait -r -e close_write,moved_to --timefmt "%s" -m "/home/vagrant" --format '%f######%e######%w##T%T' -q | ./process.sh
test.txt######CLOSE_WRITE,CLOSE######/home/vagrant/
Skipping this event as it happend within 2 seconds. TimeDiff=2
test.txt######CLOSE_WRITE,CLOSE######/home/vagrant/
This was done by using touch test.txt, multiple time every second. And as you can see second even was skipped. The process.sh is a simple bash script
#!/bin/bash
LAST_EVENT=
LAST_EVENT_TIME=0
while read line
do
DEL="##T"
EVENT_TIME=$(echo "$line" | awk -v delimeter="$DEL" '{split($0,a,delimeter)} END{print a[2]}')
EVENT=$(echo "$line" | awk -v delimeter="$DEL" '{split($0,a,delimeter)} END{print a[1]}')
TIME_DIFF=$(( $EVENT_TIME - $LAST_EVENT_TIME))
if [[ "$EVENT" == "$LAST_EVENT" ]]; then
if [[ $TIME_DIFF -gt 2 ]]; then
echo "$EVENT"
else
echo "Skipping this event as it happend within 2 seconds. TimeDiff=$TIME_DIFF"
fi
else
echo $EVENT
LAST_EVENT_TIME=$EVENT_TIME
fi
LAST_EVENT=$EVENT
done < "${1:-/dev/stdin}"
In your actual script you will disable the echo in if, this one was just for demo purpose

shell script for checking new files [duplicate]

I want to run a shell script when a specific file or directory changes.
How can I easily do that?
You may try entr tool to run arbitrary commands when files change. Example for files:
$ ls -d * | entr sh -c 'make && make test'
or:
$ ls *.css *.html | entr reload-browser Firefox
or print Changed! when file file.txt is saved:
$ echo file.txt | entr echo Changed!
For directories use -d, but you've to use it in the loop, e.g.:
while true; do find path/ | entr -d echo Changed; done
or:
while true; do ls path/* | entr -pd echo Changed; done
I use this script to run a build script on changes in a directory tree:
#!/bin/bash -eu
DIRECTORY_TO_OBSERVE="js" # might want to change this
function block_for_change {
inotifywait --recursive \
--event modify,move,create,delete \
$DIRECTORY_TO_OBSERVE
}
BUILD_SCRIPT=build.sh # might want to change this too
function build {
bash $BUILD_SCRIPT
}
build
while block_for_change; do
build
done
Uses inotify-tools. Check inotifywait man page for how to customize what triggers the build.
Use inotify-tools.
The linked Github page has a number of examples; here is one of them.
#!/bin/sh
cwd=$(pwd)
inotifywait -mr \
--timefmt '%d/%m/%y %H:%M' --format '%T %w %f' \
-e close_write /tmp/test |
while read -r date time dir file; do
changed_abs=${dir}${file}
changed_rel=${changed_abs#"$cwd"/}
rsync --progress --relative -vrae 'ssh -p 22' "$changed_rel" \
usernam#example.com:/backup/root/dir && \
echo "At ${time} on ${date}, file $changed_abs was backed up via rsync" >&2
done
How about this script? Uses the 'stat' command to get the access time of a file and runs a command whenever there is a change in the access time (whenever file is accessed).
#!/bin/bash
while true
do
ATIME=`stat -c %Z /path/to/the/file.txt`
if [[ "$ATIME" != "$LTIME" ]]
then
echo "RUN COMMNAD"
LTIME=$ATIME
fi
sleep 5
done
Check out the kernel filesystem monitor daemon
http://freshmeat.net/projects/kfsmd/
Here's a how-to:
http://www.linux.com/archive/feature/124903
As mentioned, inotify-tools is probably the best idea. However, if you're programming for fun, you can try and earn hacker XPs by judicious application of tail -f .
Just for debugging purposes, when I write a shell script and want it to run on save, I use this:
#!/bin/bash
file="$1" # Name of file
command="${*:2}" # Command to run on change (takes rest of line)
t1="$(ls --full-time $file | awk '{ print $7 }')" # Get latest save time
while true
do
t2="$(ls --full-time $file | awk '{ print $7 }')" # Compare to new save time
if [ "$t1" != "$t2" ];then t1="$t2"; $command; fi # If different, run command
sleep 0.5
done
Run it as
run_on_save.sh myfile.sh ./myfile.sh arg1 arg2 arg3
Edit: Above tested on Ubuntu 12.04, for Mac OS, change the ls lines to:
"$(ls -lT $file | awk '{ print $8 }')"
Add the following to ~/.bashrc:
function react() {
if [ -z "$1" -o -z "$2" ]; then
echo "Usage: react <[./]file-to-watch> <[./]action> <to> <take>"
elif ! [ -r "$1" ]; then
echo "Can't react to $1, permission denied"
else
TARGET="$1"; shift
ACTION="$#"
while sleep 1; do
ATIME=$(stat -c %Z "$TARGET")
if [[ "$ATIME" != "${LTIME:-}" ]]; then
LTIME=$ATIME
$ACTION
fi
done
fi
}
Quick solution for fish shell users who wanna track a single file:
while true
set old_hash $hash
set hash (md5sum file_to_watch)
if [ $hash != $old_hash ]
command_to_execute
end
sleep 1
end
replace md5sum with md5 if on macos.
Here's another option: http://fileschanged.sourceforge.net/
See especially "example 4", which "monitors a directory and archives any new or changed files".
inotifywait can satisfy you.
Here is a common sample for it:
inotifywait -m /path -e create -e moved_to -e close_write | # -m is --monitor, -e is --event
while read path action file; do
if [[ "$file" =~ .*rst$ ]]; then # if suffix is '.rst'
echo ${path}${file} ': '${action} # execute your command
echo 'make html'
make html
fi
done
Suppose you want to run rake test every time you modify any ruby file ("*.rb") in app/ and test/ directories.
Just get the most recent modified time of the watched files and check every second if that time has changed.
Script code
t_ref=0; while true; do t_curr=$(find app/ test/ -type f -name "*.rb" -printf "%T+\n" | sort -r | head -n1); if [ $t_ref != $t_curr ]; then t_ref=$t_curr; rake test; fi; sleep 1; done
Benefits
You can run any command or script when the file changes.
It works between any filesystem and virtual machines (shared folders on VirtualBox using Vagrant); so you can use a text editor on your Macbook and run the tests on Ubuntu (virtual box), for example.
Warning
The -printf option works well on Ubuntu, but do not work in MacOS.

Understanding a docker entrypoint script

The script is located here: https://github.com/docker-library/ghost/blob/master/docker-entrypoint.sh
#!/bin/bash
set -e
if [[ "$*" == npm*start* ]]; then
baseDir="$GHOST_SOURCE/content"
for dir in "$baseDir"/*/ "$baseDir"/themes/*/; do
targetDir="$GHOST_CONTENT/${dir#$baseDir/}"
mkdir -p "$targetDir"
if [ -z "$(ls -A "$targetDir")" ]; then
tar -c --one-file-system -C "$dir" . | tar xC "$targetDir"
fi
done
if [ ! -e "$GHOST_CONTENT/config.js" ]; then
sed -r '
s/127\.0\.0\.1/0.0.0.0/g;
s!path.join\(__dirname, (.)/content!path.join(process.env.GHOST_CONTENT, \1!g;
' "$GHOST_SOURCE/config.example.js" > "$GHOST_CONTENT/config.js"
fi
ln -sf "$GHOST_CONTENT/config.js" "$GHOST_SOURCE/config.js"
chown -R user "$GHOST_CONTENT"
set -- gosu user "$#"
fi
exec "$#"
From what I know, it says that if you use some variation of npm start to move some files around from $GHOST_SOURCE to $GHOST_CONTENT, do something to the config.js file, link the config file, set ownership of the content files, and then execute npm start as the user user. Otherwise, it just runs your commands normally.
The specifics are what are hard for me to understand because there are a lot of things from bash that I've never seen before. So I have a lot of questions.
for dir in "$baseDir"/*/ "$baseDir"/themes/*/; do
In the above, why do they specify both /*/ and /themes/*/? Shouldn't /*/ contain themes? Is * not a wildcard for some reason?
targetDir="$GHOST_CONTENT/${dir#$baseDir/}"
In the above, what is the point of # in the variable expansion?
tar -c --one-file-system -C "$dir" . | tar xC "$targetDir"
In the above, does this somehow save time? Why not use something like rsync? I understand the point of -C, but why -c and --one-file-system?
sed -r '
s/127\.0\.0\.1/0.0.0.0/g;
s!path.join\(__dirname, (.)/content!path.join(process.env.GHOST_CONTENT, \1!g;
' "$GHOST_SOURCE/config.example.js" > "$GHOST_CONTENT/config.js"
What does this sed command do? I know it's a replacement, but why the "$GHOST_SOURCE/config.example.js" > "$GHOST_CONTENT/config.js" as the end?
ln -sf "$GHOST_CONTENT/config.js" "$GHOST_SOURCE/config.js"
In the above, what is the point of this symlink? Why try to link them to each other if both files already exist?
set -- gosu user "$#"
In the above what does calling set with no args do?
I hope that's not too much. I felt making a separate question for each of these would be too much especially since it's all related to each other.
for dir in "$baseDir"/*/ "$baseDir"/themes/*/; do
In the above, why do they specify both /*/ and /themes/*/? Shouldn't
/*/ contain themes? Is * not a wildcard for some reason?
themes/ is in the first match, but themes/*/ is not, so you need the second entry to include the contents of themes.
targetDir="$GHOST_CONTENT/${dir#$baseDir/}"
In the above, what is the point of # in the variable expansion?
It removes the $baseDir prefix from $dir. So for example:
bash$ dir=/home/bmitch/data/docker
bash$ echo $dir
/home/bmitch/data/docker
bash$ echo ${dir#/home/bmitch}
/data/docker
tar -c --one-file-system -C "$dir" . | tar xC "$targetDir"
In the above, does this somehow save time? Why not use something like
rsync? I understand the point of -C, but why -c and --one-file-system?
rsync may not be installed on every machine by default, tar is fairly universal. The -c is to create, vs extract, and --one-file-system avoids tar continuing to an outside mount point (nfs, symlink to root, etc).
sed -r '
s/127\.0\.0\.1/0.0.0.0/g;
s!path.join\(__dirname, (.)/content!path.join(process.env.GHOST_CONTENT, \1!g;
' "$GHOST_SOURCE/config.example.js" > "$GHOST_CONTENT/config.js"
What does this sed command do? I know it's a replacement, but why the
"$GHOST_SOURCE/config.example.js" > "$GHOST_CONTENT/config.js" as the
end?
config.example.js is the input (last arg to the sed), config.js is the output (after the >). So it takes the config.example.js, change the ip address from 127.0.0.1 to 0.0.0.0, effectively listening on all interfaces/ip's instead of just internally on the loopback. The second half of the sed is changing the path.join arguments from __dirname to process.env.GHOST_CONTENT.
ln -sf "$GHOST_CONTENT/config.js" "$GHOST_SOURCE/config.js"
In the above, what is the point of this symlink? Why try to link them
to each other if both files already exist?
The $GHOST_SOURCE/config.js is replaced (-f) with a link to $GHOST_CONTENT/config.js. Symbolic links give a file name reference to another actual file, so there will be two names, but one copy of the data, which means you will only have a single configuration in this situation.
set -- gosu user "$#"
In the above what does calling set with no args do?
This changes the values of $1, $2, ... $n to be $1=gosu, $2=user, $3=the old $1, $4=the old $2..., essentially adding the gosu and user to the beginning of the passed parameters to the script. The -- makes sure that set doesn't interpret any values from $# as a flag for itself.

Bash script for inotifywait - How to write deletes to log file, and cp close_writes?

I have this bash script:
#!/bin/bash
inotifywait -m -e close_write --exclude '\*.sw??$' . |
#adding --format %f does not work for some reason
while read dir ev file; do
cp ./"$file" zinot/"$file"
done
~
Now, how would I have it do the same thing but also handle deletes by writing the filenames to a log file?
Something like?
#!/bin/bash
inotifywait -m -e close_write --exclude '\*.sw??$' . |
#adding --format %f does not work for some reason
while read dir ev file; do
# if DELETE, append $file to /inotify.log
# else
cp ./"$file" zinot/"$file"
done
~
EDIT:
By looking at the messages generated, I found that inotifywait generates CLOSE_WRITE,CLOSE whenever a file is closed. So that is what I'm now checking in my code.
I tried also checking for DELETE, but for some reason that section of the code is not working. Check it out:
#!/bin/bash
fromdir=/path/to/directory/
inotifywait -m -e close_write,delete --exclude '\*.sw??$' "$fromdir" |
while read dir ev file; do
if [ "$ev" == 'CLOSE_WRITE,CLOSE' ]
then
# copy entire file to /root/zinot/ - WORKS!
cp "$fromdir""$file" /root/zinot/"$file"
elif [ "$ev" == 'DELETE' ]
then
# trying this without echo does not work, but with echo it does!
echo "$file" >> /root/zinot.txt
else
# never saw this error message pop up, which makes sense.
echo Could not perform action on "$ev"
fi
done
In the dir, I do touch zzzhey.txt. File is copied. I do vim zzzhey.txt and file changes are copied. I do rm zzzhey.txt and the filename is added to my log file zinot.txt. Awesome!
You need to add -e delete to your monitor, otherwise DELETE events won't be passed to the loop. Then add a conditional to the loop that handles the events. Something like this should do:
#!/bin/bash
inotifywait -m -e delete -e close_write --exclude '\*.sw??$' . |
while read dir ev file; do
if [ "$ev" = "DELETE" ]; then
echo "$file" >> /inotify.log
else
cp ./"$file" zinot/"$file"
fi
done

Resources