Find all files that need updating - bash

I use the script below to convert all pdf files in a directory to png files, and I want to run it only over the files that have yet to be converted.
#!/bin/bash
# Convert pdf to png
for f in *.pdf
do
echo "Converting $f"
gs -dNOPAUSE -dBATCH -sDEVICE=png256 -r480 -q -sOutputFile="png/$f.png" "$f"
done
rename -f 's/\.pdf\.png/\.png/' png/*.pdf.png
How do I modify the loop so that it is restricted to files where the corresponding png file either does not exist or is older than the pdf file?

A simple modification to your script:
#!/bin/bash
# Convert pdf to png
for f in *.pdf
do
png="png/${f%pdf}png"
if [ -e "$png" -a "$f" -nt "$png" ]; then
continue
fi
echo "Converting $f"
gs -dNOPAUSE -dBATCH -sDEVICE=png256 -r480 -q -sOutputFile="$png" "$f"
done
Firstly, we create a $png variable using in place editing. Basically the %pdf tells bash to remove the last occurrence of pdf, which is the extension. Then we can place this into a string that prefixes it with png/ and adds the png extension. This saves you the last rename command.
Now we have an if statement that continues the loop if a certain condition is met. Continue means go to the next iteration of the loop, without executing anything else for this iteration. The condition is "$png" exists (-e "$png") and (-a) "$f" is newer than "$png" ("$f" -nt "$png").
But I would suggest writing a Makefile:
PDFS := $(wildcard *.pdf)
PNGS := $(addprefix png/,$(PDFS:.pdf=.png))
all: $(PNGS)
png/%.png: %.pdf
gs -dNOPAUSE -dBATCH -sDEVICE=png256 -r480 -q -sOutputFile="$#" "$<"
And run it with make any time you want to generate newer PNGs.

Related

I'm using Pandoc to convert Markdown to .docx - how do I remove the .md from the resulting filename?

I've been using a shell script in Automator on MacOS (OSX) successfully, but my method retains the '.md' extension in the resulting filename.
For example, if I input the file myfile.md the output is myfile.md.docx
This is my script:
for f in "$#"
do
if [[ "$f" = *.md ]]; then
/Users/myname/opt/anaconda3/bin/pandoc -o "${f%}.docx" -f markdown -t docx $f && open "${f%}.docx"
fi
done
Can anyone help me with this last step?
Use -o "${f%.*}.docx" to remove the original extension.

Bash with recursion into subfolders containing spaces

I am trying to add unique IDs to the pictures taken with my DSLR. The DSLR saves both a RAW file (NEF) and an actual image file (JPG). These pairs represents the same image, and should therefore have the same image ID.
I tried the following bash script which is kind of working. However, since there are spaces in my folder names, I have to run the script from each subfolder instead of from the parent pictures folder.
How do I redo the script, so it would allow subfolder whose names contains spaces?
#!/bin/bash
LIB="."
for file in $(find $LIB -name '*.NEF'); do
UUID=`uuidgen`
exiftool -q -if 'not $ImageUniqueID' "$file" -ImageUniqueID=$UUID -overwrite_original;
exiftool -q -if 'not $ImageUniqueID' "${file%.NEF}.JPG" -ImageUniqueID=$UUID -overwrite_original;
done
Use a loop or find, not both.
# With a loop
shopt -s globstar
for f in "$LIB"/**/*.NEF; do
uuid=$(uuidgen)
exiftool -q -if 'not $ImageUniqueID' "$file" -ImageUniqueID=$uuid -overwrite_original
exiftool -q -if 'not $ImageUniqueID' "${file%.NEF}.JPG" -ImageUniqueID=$uuid -overwrite_original
done
# With find
find "$LIB" -name "*.NEF" -exec sh -c '
uuid=$(uuidgen)
exiftool -q -if "not \$ImageUniqueID" "$1" -ImageUniqueID=$uuid -overwrite_original;
exiftool -q -if "not \$ImageUniqueID" "${1%.NEF}.JPG" -ImageUniqueID=$uuid -overwrite_original
' _ {} \;
The startup time of exiftool is its biggest performance hit and running once for each file will significantly increase the run time, especially when processing hundreds, if not thousands, of files.
Unless you specifically need the type of id that uuidgen generates, exiftool has to the ability to create a unique id with the NewGUID tag. As listed on the Extra Tags page, it consists of "a new, random GUID with format YYYYmmdd-HHMM-SSNN-PPPP-RRRRRRRRRRRR, where Y=year, m=month, d=day, H=hour, M=minute, S=second, N=file sequence number in hex, P=process ID in hex, and R=random hex number". A hashtag can be added to the end of the tag name (e.g. NewGUID#) to suppress the dashes.
You could then run
exiftool -overwrite_original -ext Nef -r -q -if 'not $ImageUniqueID' '-ImageUniqueID<NewGUID' .
to write all the nef files recursively (enabled by the -r option) and then run a second command to copy the ImageUniqueID from the nefs to the jpgs with
exiftool -overwrite_original -ext jpg -r -q -TagsFromFile %d%f.nef -ImageUniqueID .

How to recursively resize images from one folder to another?

I would like to create thumbnails of images I have organised into a set of nested subdirectories into a mirror of the file structure so that a command of the type:
./imageresize.sh Large Small 10
...would convert any .jpg or .JPG files, in the directories nested under "./Large":
./Large/Holidays/001.jpg
./Large/Holidays/002.jpg
./Large/Holidays/003.jpg
./Large/Pets/Dog/001.jpg
./Large/Pets/Dog/002.jpg
./Large/Pets/Cat/001.jpg
into thumbnails of 10% to a mirror destination with a different top directory ("Small" instead of "Large" in this e.g.):
./Small/Holidays/001.jpg
./Small/Holidays/002.jpg
./Small/Holidays/003.jpg
./Small/Pets/Dog/001.jpg
./Small/Pets/Dog/002.jpg
./Small/Pets/Cat/001.jpg
This is what I have so far, but I can't seem to get it working. The $newfile variable seems invalid but I don't know why, and when testing, it outputs the result of the 'convert' command to the screen. Any help/suggestions greatly appreciated.
#!/bin/bash
#manage whitespace and escape characters
OIFS="$IFS"
IFS=$'\n'
#create file list
filelist=$(find ./$1/ -name "*.jpg" -o -name "*.JPG")
for file in $filelist
do
#create destination path for 'convert' command
newfile= ${file/$1/$2}
convert "$file" -define jpeg:extent=10kb -scale $3% "$newfile"
done
Don't know if it's just a copy/paste error or if it's actually in your script, but this line:
newfile= ${file/$1/$2}
would be an invalid assignment in Bash as spaces around = isn't allowed when assigning.
Try this instead:
newfile=${file/$1/$2}
As a side-note. find has a case-insensitive search too, -iname, so you could do:
filelist=$(find ./$1/ -iname "*.jpg")
It also has -exec for executing commands on the result set. It's explained very well here: https://unix.stackexchange.com/questions/12902/how-to-run-find-exec So you may be able to do it in one single find. (Note: That's not necessarily better, in most cases it's just a matter of preference, I just mention it as a possibility.)
So - revised working script with corrections suggested by madsen and 4ae1e1 (thanks both), and with rsync commands to create directory structure first, (then clean extraneous files from the destination) :). I've added an extra parameter and parameter checker so that now you can specify source, destination, approx destination file size in kb and percentage of original. Hope it helps someone else. :)
#!/bin/bash
#manage whitespace and escape characters
OIFS="$IFS"
IFS=$'\n'
#check parameters
if [ $# -lt 4 ] || [ "$1" = "--help" ]
then # if no parameters or '--help' as $1 - show help
echo "______________________________________________________________"
echo ""
echo "Useage: thumbnailmirror [Source] [Destination] [Filesize] [Percentage]"
echo "Source - e.g. 'Fullsize' (directory must exist)"
echo "Destination - e.g. 'Thumnail' (directory must exist)"
echo "Filesize - approx filesize in kb e.g. '10'"
echo "Percentage - % of reduction (1-100)"
echo "e.g. thumbnailmirror Fullsize Thumnail 18 7"
echo "______________________________________________________________"
else # parameters exist
#syncronise directory structure (directories only)
rsync -a --include '*/' --exclude '*' ./$1/ ./$2/
# delete any extraneous files and directories at destination
rsync -a --delete --existing --ignore-existing ./$1/ ./$2/
#create file list ( -iname means not case sensitive)
filelist=$(find ./$1/ -iname "*.jpg")
for file in $filelist
do
#define destination filename for 'convert' command
newfile=${file/$1/$2}
if [ ! -f "$newfile" ] #if file doesn't exists create it
then
convert "$file" -define jpeg:extent=$3kb -quality 100 -scale $4% "$newfile"
echo "$file resized"
else #skip it
echo "Skipping $file - exists already"
fi
done
fi

HandBrakeCLI bash script convert all videos in a folder

Firstly, I searched around for my problem. But none can solve it.
I want to convert all videos file in a directory and the output will be saved in another directory. I got a bash script from somewhere I dont remember.
#!/bin/bash
SRC="/home/abc/public_html/filex/store/vids/toriko/VIDEOS HERE"
DEST="/home/abc/public_html/filex/store/vids/toriko/51-100"
DEST_EXT=mp4
HANDBRAKE_CLI=HandBrakeCLI
PRESET="iPhone & iPod Touch"
for FILE in "`ls $SRC`"
do
filename=$(basename $FILE)
extension=${filename##*.}
filename=${filename%.*}
$HANDBRAKE_CLI -i "$SRC"/$FILE -o "$DEST"/"$filename".$DEST_EXT "$PRESET"
done
the problem is, the output of the file will be without filename.. only ".mp4".
and, there is only 1 file generated.. means, from 50 videos in the folder, only 1 files generated with name ".mp4" and after that, HandBrakeCLI exit.
can anyone fix my code?
I got no experince in bash coding.. so, the right script giiven will be appreciate :)
Your line
for FILE in "`ls $SRC`"
effectively creates only one iteration where FILE contains the list of the files (and it is not able to handle the space in $SRC). Better replace it with
for FILE in "$SRC"/*
Example:
$ ls test
1.txt 2.txt
$ SRC=test; for f in "`ls $SRC`" ; do echo $f; done
1.txt 2.txt
$ SRC=test; for f in "$SRC"/* ; do echo $f; done
test/1.txt
test/2.txt
Side note: you can have a space in there with no problem
$ ls "the test"
1.txt 2.txt
$ SRC="the test"; for f in "$SRC"/* ; do echo $f; done
the test/1.txt
the test/2.txt
I tried this script, and others like it, but I wanted to convert recursive directory tree's and have files placed in the same directory with .mp4 extension and delete .avi files, after much trial and error I gave up on this code and searched for a new code, id like to credit
http://www.surlyjake.com/blog/2010/08/10/script-to-run-handbrake-recursively-through-a-folder-tree/
For the original code!
Here is my modified script, barely modified BTW this script is short, sweet and easy to understand.
#!/bin/bash
# This Script Goes in Root Folder of TV show -- Example Folder Structure
# /Stargate/Season\ 1/Epiosde.avi
# /Stargate/Season\ 2/Epiosde.avi
# /Stargate/handbrake_folder.script
# Outputs all Files back inside same dir's and does all folders inside Startgate DIR
# /Stargate/Season\ 1/Epiosde.mp4
# /Stargate/Season\ 2/Epiosde.mp4
# PRESET = -o flags for CLI can be got from GUI under Activity Log or from https://trac.handbrake.fr/wiki/CLIGuide OR you can use actual Presets!
# PRESET="iPhone & iPod Touch"
PRESET="--modulus 2 -e x264 -q 20 --vfr -a 1 -E ac3 -6 5point1 -R Auto -B 384 -D 0 --gain 0 --audio-fallback ac3 --encoder-preset=veryfast --encoder-level="5.2" --encoder-profile=high --verbose=1"
if [ -z "$1" ] ; then
TRANSCODEDIR="."
else
TRANSCODEDIR="$1"
fi
find "$TRANSCODEDIR"/* -type f -name "*.avi" -exec bash -c 'HandBrakeCLI -i "$1" -o "${1%\.*}".mp4 --preset="$PRESET"' __ {} \; && find . -name '*.avi' -exec rm -r {} \;
BE WARNED: THIS WILL CONVERT THEN DELETE ALL .AVI FILES ABOVE THE SCRIPT IN FILE TREE!
Feel free to remove the
[-name "*.avi"] & [&& find . -name '*.avi' -exec rm -r {} \;]
to disable only converting .avi and removal of .avi or modify to suite another extension.
I have found the solution:
#!/bin/bash
SRC="/home/abc/public_html/filex/store/vids/toriko/VIDEOS HERE"
DEST="/home/abc/public_html/filex/store/vids/toriko/51-100"
DEST_EXT=mp4
HANDBRAKE_CLI=HandBrakeCLI
for FILE in "$SRC"/*
do
filename=$(basename "$FILE")
extension=${filename##*.}
filename=${filename%.*}
$HANDBRAKE_CLI -i "$FILE" -o "$DEST"/"$filename".$DEST_EXT
done
I just tried using this script with the modification suggested above. I found I need to to put double quotes around the two uses of $FILE in order to handle file names with spaces.
So...
filename=$(basename "$FILE")
and
$HANDBRAKE_CLI -i "$SRC"/"$FILE" -o "$DEST"/"$filename".$DEST_EXT "$PRESET"
I'd rather prefer this solution:
#!/bin/bash
SRC="$1"
DEST="$2"
EXT='mp4'
PRESET='iPhone & iPod Touch'
#for FILE in "`ls $SRC`"; do
for FILE in `find . -type f`; do
FILE=$(basename "$FILE")
filename=$(basename "$FILE")
extension=${filename##*.}
filename=${filename%.*}
HandBrakeCLI -i "$SRC"/$FILE -o "$DEST"/"$filename"."$EXT" "$PRESET"
done

Batch converting videos in folders and sub folders

I have a huge collection of videos that all need to be converted into mp4. The folder structure is like this
Events
Chicago
Boston
San Fran
London Expo
Inside each event holds all of videos in either avi format or .mkv. I want them to be converted to the same file name. but with the mp4 extension.
My question is how do I loop through a folders sub folders, and also keep the file names because at the moment they have spaces in them.. Here is what I have at the moment.
sourcedir="$1"
destdir="$2"
cd "$sourcedir"
for i in `ls`; do
HandBrakeCLI -i "$i" -o "$destdir/${i%.*}.mp4" --preset="AppleTV"
echo $i
done
Phillips Code:
cd "$sourcedir"
echo "Directory: $sourcedir"
destdir = sourcedir
for subdir in *
do
if [[ -d "$subdir" ]]
then
for file in "$subdir"/*
do
HandBrakeCLI -i "$file" -o "${file%.*}.mp4" --preset=AppleTV
echo "$file"
done
fi
done
Use a nested loop, and don't use ls:
for subdir in "$sourcedir"/*
do
if [[ -d "$subdir" ]]
then
for file in "$subdir"/*
do
HandBrakeCLI -i "$file" -o "$destdir/${file%.*}.mp4" --preset=AppleTV
echo "$file"
done
fi
done
Another option is to use find:
find "$sourcedir" -maxdepth 2 -mindepth 2 -type f -exec bash -c 'HandBrakeCLI -i "$0" -o "'"$destdir"'/${0%.*}.mp4" --preset=AppleTV' '{}' ';' -print
Both solutions will work with filenames containing spaces or newlines.
Either use for i in "$sourcedir/*" (or since you've already done a cd there you could do for i in *).
or do find "$sourcedir" -type f | while read -r i (with this, the variable i will include the source directory name, so you'll have to strip that off using a brace expansion or basename).
change program to iVI link to iVi.
this program does the trcik, even better than handbrake...
bash 4
shopt -s globstar
destdir="/somewhere"
for file in **/*.mkv **/*.avi
do
HandBrakeCLI -i "$file" -o "$destdir/${file%.*}.mp4" --preset=AppleTV
done

Resources