How do I list newest directory and add as variable to bash script to process files recursively - bash

How do I list newest directory and add as variable to bash script to process files recursively
ls -t1 | head -n1
Works perfectly to list the latest directory, but I want to add that directory name to my script so I can process the files within using the following script:
#!/bin/bash
ls | while read -r FILE
do
mv -v "$FILE" `echo $FILE | tr ' ' '_' `
done
ls | while read -r FILE
do
mv -v "$FILE" `echo $FILE | tr '\*.JPEG' '\*.jpg' `
done
mogrify -resize 750 *.jpg
wait
jpegoptim *.jpg –max=70 --strip-all
exit
I also want to process the files recursively, there might be at most one level of sub directories.
Basically keep the bash script at the root of the directory and process all latest directories and sub directories files.
OK I modified the script to this:
#!/bin/bash
DIR=ls -t1 | head -n1
ls $DIR | while read -r FILE
do
mv -v "$FILE" `echo $FILE | tr ' ' '_' `
done
ls $DIR | while read -r FILE
do
mv -v "$FILE" `echo $FILE | tr '\*.JPEG' '\*.jpg' `
done
mogrify -resize 750 $DIR/*.jpg
wait
jpegoptim $DIR/*.jpg –max=70 --strip-all
exit
But it does not seem to recognise the $DIR variable.

This bash script will rename and convert the jpg files in the newest directory in the current directory and the files in the first level of directories under that directory.
#!/bin/bash
FIRST_DIR=`ls -t1F | grep / | head -n1`
DIR="./${FIRST_DIR}"
ls -t1F $DIR | while read -r FILE
do
if [ "$FILE" ]
then
if [[ $FILE = */ ]]
then
echo "here ${DIR}${FILE}."
DEEP_DIR="${DIR}${FILE}"
ls -t1 $DEEP_DIR | while read -r FILE2
do
if [ "$FILE2" ]
then
if [[ $FILE2 != */ ]]
then
RENAME=`echo ${FILE2//\*/} | tr ' ' '_' `
mv -v "${DEEP_DIR}${FILE2//\*/}" "${DEEP_DIR}${RENAME}"
FILE2=$RENAME
RENAME2=`echo ${FILE2//\*/} | tr '\*.JPEG' '\*.jpg' `
mv -v "${DEEP_DIR}${FILE2//\*/}" "${DEEP_DIR}${RENAME2}"
FILE2=$RENAME2
fi
fi
done
mogrify -resize 750 "$DEEP_DIR*.jpg"
wait
jpegoptim "$DEEP_DIR*.jpg" –max=70 --strip-all
fi
if [[ $FILE != */ ]]
then
RENAME=`echo ${FILE//\*/} | tr ' ' '_' `
mv -v "${DIR}${FILE//\*/}" "${DIR}${RENAME}"
FILE=$RENAME
RENAME2=`echo ${FILE//\*/} | tr '\*.JPEG' '\*.jpg' `
mv -v "${DIR}${FILE//\*/}" "${DIR}${RENAME2}"
FILE=$RENAME2
fi
fi
done
if [ "$FIRST_DIR" ]
then
mogrify -resize 750 "$DIR*.jpg"
wait
jpegoptim "$DIR*.jpg" –max=70 --strip-all
fi
Here are a couple of good links about bash programming:
http://tldp.org/LDP/abs/html/comparison-ops.html
http://tldp.org/LDP/abs/html/string-manipulation.html

Related

improve bash script to add one more action if success

I have a bash script to find out all non empty files under some directory. currently it only print the file name if found.
I would like to add one more line to print first 32 bytes in hex format.
#!/bin/sh
files=$(find /data/ -type f ! -empty)
for f in $files;
do
if [ -f "$f" ]; then
tr -d '\000' <$f | tr -c '\000' '\n' | grep -q -m 1 ^ || echo $f
fi
done
I try to add one more "&& xxd -g 1 -l 32 $f" at the end but it doesn't work!
Get the first 32 chars from the file:
dd if=so.bash ibs=32 count=1 2>/dev/null | od -h
dd gets the first 32 chars
od -h prints them in hex format
You could do it with xxd as well
xxd -l 32 $f
where $f is the file
#!/bin/bash
files=$(find /data/ -type f ! -empty)
for f in $files
do
if [ -f "$f" ]; then
tr -d '\000' <"$f" | tr -c '\000' '\n' | grep -q -m 1 ^ || echo $f
xxd -l 32 "$f"
echo ""
fi
done
the echo "" is to have an empty line between each file to split the output.
Suggesting one line gawk script:
gawk 'BEGINFILE{print FILENAME}/[^\x00]/{system("xxd -l 32 "FILENAME;nextfile}' $(find /data/ -type f ! -empty)
gawk Explanation
BEGINFILE{print FILENAME}
BEGINFILE{ # before processing a file
print FILENAME; # print the filename
}
/[^\x00]/{system("xxd -l 32 "FILENAME);nextfile}
/[^\x00]/{ # if records contains non NULL character
system("xxd -l 32 "FILENAME); # print first 32 hex charachters
nextfile; # read next file
}
Every command run in bash shell returns a value that’s stored in the bash variable “$?”.
Then the command can be split as below:
#!/bin/sh
files=$(find /data/ -type f ! -empty)
for f in $files;
do
if [ -f "$f" ]; then
tr -d '\000' <$f | tr -c '\000' '\n' | grep -q -m 1 ^
if [ $? -ne 0 ]; then
echo $f
xxd -l 32 $f
fi
fi
done

Looping through each file in directory - bash

I'm trying to perform certain operation on each file in a directory but there is a problem with order it's going through. It should do one file at the time. The long line (unzipping, grepping, zipping) works fine on a single file without a script, so there is a problem with a loop. Any ideas?
Script should grep through through each zipped file and look for word1 or word2. If at least one of them exist then:
unzip file
grep word1 and word2 and save it to file_done
remove unzipped file
zip file_done to /donefiles/ with original name
remove file_done from original directory
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c 'word1\|word2' $file)
if [[ $counter -gt 0 ]]; then
echo $counter
for file in *.gz; do
filenoext=${file::-3}
filedone=${filenoext}_done
echo $file
echo $filenoext
echo $filedone
gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip -f -c $filedone > /donefiles/$file | rm -f $filedone
done
else
echo "nothing to do here"
fi
done
The code snipped you've provided has a few problems, e.g. unneeded nested for cycle and erroneous pipeline
(the whole line gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip...).
Note also your code will work correctly only if *.gz files don't have spaces (or special characters) in names.
Also zgrep -c 'word1\|word2' will also match strings like line_starts_withword1_orword2_.
Here is the working version of the script:
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c -E 'word1|word2' $file) # now counter is the number of word1/word2 occurences in $file
if [[ $counter -gt 0 ]]; then
name=$(basename $file .gz)
zcat $file | grep -E 'word1|word2' > ${name}_done
gzip -f -c ${name}_done > /donefiles/$file
rm -f ${name}_done
else
echo 'nothing to do here'
fi
done
What we can improve here is:
since we unzipping the file anyway to check for word1|word2 presence, we may do this to temp file and avoid double-unzipping
we don't need to count how many word1 or word2 is inside the file, we may just check for their presence
${name}_done can be a temp file cleaned up automatically
we can use while cycle to handle file names with spaces
#!/bin/bash
tmp=`mktemp /tmp/gzip_demo.XXXXXX` # create temp file for us
trap "rm -f \"$tmp\"" EXIT INT TERM QUIT HUP # clean $tmp upon exit or termination
find . -maxdepth 1 -mindepth 1 -type f -name '*.gz' | while read f; do
# quotes around $f are now required in case of spaces in it
s=$(basename "$f") # short name w/o dir
gunzip -f -c "$f" | grep -P '\b(word1|word2)\b' > "$tmp"
[ -s "$tmp" ] && gzip -f -c "$tmp" > "/donefiles/$s" # create archive if anything is found
done
It looks like you have an inner loop inside the outer one :
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c 'word1\|word2' $file)
if [[ $counter -gt 0 ]]; then
echo $counter
for file in *.gz; do #<<< HERE
filenoext=${file::-3}
filedone=${filenoext}_done
echo $file
echo $filenoext
echo $filedone
gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip -f -c $filedone > /donefiles/$file | rm -f $filedone
done
else
echo "nothing to do here"
fi
done
The inner loop goes through all the files in the directory if one of them contains file1 or file2. You probably want this :
#!/bin/bash
for file in *.gz; do
counter=$(zgrep -c 'word1\|word2' $file)
if [[ $counter -gt 0 ]]; then
echo $counter
filenoext=${file::-3}
filedone=${filenoext}_done
echo $file
echo $filenoext
echo $filedone
gunzip $file | grep 'word1\|word2' $filenoext > $filedone | rm -f $filenoext | gzip -f -c $filedone > /donefiles/$file | rm -f $filedone
else
echo "nothing to do here"
fi
done

Bash : Find and Remove duplicate files from different folders

I have two folders with some common files, I want to delete duplicate files from xyz folder.
folder1:
/abc/file1.csv
/abc/file2.csv
/abc/file3.csv
/abc/file4.csv
folder2:
/xyz/file1.csv
/xyz/file5.csv
I want to compare both folders and remove duplicate from /xyz folder. Output should be: file5.csv
For now I am using :
find "/xyz" "/abc" "/abc" -printf '%P\n' | sort | uniq -u | -exec rm {} \;
But it failing with reason : if -exec is not a typo you can run the following command to lookup the package that contains the binary:
command-not-found -exec
-bash: -exec: command not found
-exec is an option to find, you've already exited the command find when you started the pipes.
Try xargs instead, it take all the data from stdin and appends to the program.
UNTESTED
find "/xyz" "/abc" "/abc" -printf '%P\n' | sort | uniq -u | xargs rm
Find every file in 234 and 123 directory get filename by -printf, sort them, uniq -d give list of duplications, give back path by sed, using 123 directory to delete the duplications from, and pass files to xargs rm
Command:
find ./234 ./123 -type f -printf '%P\n' | sort | uniq -d | sed 's/^/.\/123\//g' | xargs rm
sed don't needed if you are in the ./123 directory and using full path for folders in find.
Another approach: just find the files in abc and attempt to remove them from xyz:
UNTESTED
find /abc -type f -printf 'rm -f /xyz/%P' | sh
Remove Duplicate Files From Particular Directory
FileList=$(ls)
for D1 in $FileList ;do
if [[ -f $D1 ]]; then
for D2 in $FileList ;do
if [[ -f $D2 ]]; then
if [[ $D1 == $D2 ]]; then
: 'Skip Orignal File'
else
if [[ $(md5sum $D1 | cut -d'=' -f 2 | cut -d ' ' -f 1 ) == $(md5sum $D2 | cut -d'=' -f 2 | cut -d ' ' -f 1 ) ]]; then
echo "Duplicate File Found : $D2"
rm -rf $D2
fi #Detect Duplicate Using MD5
fi #Skip Orginal File
fi #D2 File available Then Next
done
fi #D1 File available Then Next
done

Converting FLAC file collection to ALAC in another directory with shell script

I have searched many forums and websites to create an ALAC collection from my FLAC collection with the same directory structure with no success. Therefore I coded my own shell script and decided to share here so others can use or improve on it.
Problems I wanted to solve:
Full automation of conversion. I did not want to go and run scripts
in each and every directory.
Recursive file search
Moving all the structure from one location to another by converting flac to alac and copying the artwork. nothing else.
I did not want flac and alac files in the same directory.(which the below
script I believe can do that)
Here is how the script turned out. It works for me, I hope it does for you as well. I am using Linux Mint and bash shell.
2014-12-08 - Made some changes and now it is working fine. Before it was creating multiple copies.
Usage is: ./FLACtoALAC.sh /sourcedirectory /targetdirectory
Here are some explanations:
Source: /a/b/c/d/e/ <- e has flac
/g/f/k <- k has artwork
/l <- l has mp3
Target: /z/u/v/g/f
when the command is run : ./FLACtoALAC.sh /a/b/ /z/u/
I want the structure look like:
/z/u/v/g/f <- f was already there
/c/d/e/ <- e had flac, so created with the tree following source (/a/b)
/c/g/f/k <- k had artwork, so created with the tree following source (/a/b)
not created l <- l did not have any of the png,jpg or flac files.
I do not want to create any directory that does not contain png, jpg or flac,
unless it is a parent to one of such those directories.
Now the updated code:
#!/bin/bash
if [[ $1 ]]
then
if [[ ${1:0:1} = / || ${1:0:1} = ~ ]]
then Source_Dir=$1
elif [[ ${1:0:1} = . ]]
then Source_Dir=`pwd`
else Source_Dir=`pwd`'/'$1
fi
else Source_Dir=`pwd`'/'
fi
if [[ $2 ]]
then
if [[ ${2:0:1} = / || ${2:0:1} = ~ ]]
then Target_Dir=$2
elif [[ ${2:0:1} = . ]]
then Target_Dir=`pwd`
else Target_Dir=`pwd`'/'$2
fi
else Target_Dir=`pwd`'/'
fi
echo "Source Directory : "$Source_Dir
echo "Target Directory : "$Target_Dir
typeset -i Source_Dir_Depth
Source_Dir_Depth=`echo $Source_Dir | grep -oi "\/" | wc -l`
typeset -i Target_Dir_Depth
Target_Dir_Depth=`echo $Target_Dir | grep -oi "\/" | wc -l`
echo "Depth of the Source Directory: "$Source_Dir_Depth
echo "Depth of the Target Directory: "$Target_Dir_Depth
echo "Let's check if the Target Directory exists, if not we will create"
typeset -i Number_of_depth_checks
Number_of_depth_checks=$Target_Dir_Depth+1
for depth in `seq 2 $Number_of_depth_checks`
do
Target_Directory_Tree=`echo ${Target_Dir} | cut -d'/' -f-${depth}`
if [[ -d "$Target_Directory_Tree" ]]
then
echo "This directory exists ("$Target_Directory_Tree"), moving on"
else
Create_Directory=`echo ${Target_Dir} | cut -d'/' -f-${depth}`
echo "Creating the directory/subdirectory $Create_Directory"
mkdir -pv "$Create_Directory"
fi
done
Directory_List=`find "${Source_Dir}" -type d -exec sh -c 'ls -tr -1 "{}" | sort | egrep -iq "*.(jpg|png|flac)$"' ';' -print`
oIFS=$IFS
IFS=$'\n'
for directories in $Directory_List
do
echo "Directories coming from the source : $directories"
typeset -i directories_depth
directories_depth=`echo $directories | grep -oi "\/" | wc -l`
echo "Number of sub-directories to be checked: $Source_Dir_Depth"
typeset -i number_of_directories_depth
number_of_directories_depth=$directories_depth+1
for depth in `seq 2 $number_of_directories_depth`
do
Source_Tree=`echo ${Source_Dir} | cut -d'/' -f-${depth}`
Subdirectory_Tree=`echo ${directories} | cut -d'/' -f-${depth}`
Subdirectory_Remaining_Tree=`echo ${directories} | cut -d'/' -f${depth}-`
echo "source tree : $Source_Tree"
echo "source tree : $Subdirectory_Tree"
if [[ $depth -le $Source_Dir_Depth && $Source_Tree = $Subdirectory_Tree ]]
then
echo "Common Directory, skipping ($Subdirectory_Tree)"
continue
else
export Targetecho=$(echo $Target_Dir | sed -e 's/\r//g')
export Destination_Directory=${Targetecho}${Subdirectory_Remaining_Tree}
echo "Destination directory is : $Destination_Directory"
export Sub_directories_depth=`echo $Destination_Directory | grep -oi "\/" | wc -l`
echo "Total destination depth : $Sub_directories_depth"
echo "Now we are checking target directory structure"
fi
break
done
echo "Gettin into the new loop to verify/create target structure"
typeset -i number_of_Sub_directories_depth
number_of_Sub_directories_depth=$Sub_directories_depth+1
for subdepth in `seq 2 $number_of_Sub_directories_depth`
do
Target_Subdirectory_Tree=`echo ${Destination_Directory} | cut -d'/' -f-${subdepth}`
if [[ $subdepth < $number_of_Sub_directories_depth && -d "$Target_Subdirectory_Tree" ]]
then
echo "Directory already exists in the destination ($Target_Subdirectory_Tree)"
elif [[ $subdepth < $number_of_Sub_directories_depth && ! -d "$Target_Subdirectory_Tree" ]]
then
echo "Creating the path in the destination ($Target_Subdirectory_Tree)"
mkdir -pv "$Target_Subdirectory_Tree"
elif [[ $subdepth -eq $number_of_Sub_directories_depth ]]
then
if [[ ! -d "$Destination_Directory" ]]
then
echo "Creating Directory: $Destination_Directory"
mkdir -pv "$Destination_Directory"
fi
echo "Directory already exists in the destination ($Destination_Directory)"
#Flac file processing starts here once the directory is found
Flac_File_List=`(shopt -s nocaseglob ; ls -tr "${directories}"/*.flac | sort)`
echo "List of files in $directories :"
echo $Flac_File_List
for flac_files in $Flac_File_List
do
echo "files : $flac_files"
typeset -i flac_file_depth
flac_file_depth=`echo $flac_files | grep -oi "\/" | wc -l`
flac_file_depth=$flac_file_depth+1
echo "flac_file_depth : $flac_file_depth"
Flac_File_Name=`echo ${flac_files} | cut -d'/' -f${flac_file_depth}`
echo "Flac_File Name : $Flac_File_Name"
Destination_File=${Destination_Directory}'/'${Flac_File_Name}
echo "will convert $Flac_File_Name from $flac_files to $Destination_File"
yes | ffmpeg -i "$flac_files" -vf "crop=((in_w/2)*2):((in_h/2)*2)" -c:a alac "${Destination_File%.flac}.m4a"
done
#Artwork file processing starts here once the directory is found
Art_File_List=`(shopt -s nocaseglob ; ls -tr "${directories}"/*.{png,jpg} | sort)`
echo "List of files in $directories :"
echo $Art_File_List
for art_files in $Art_File_List
do
echo "files : $art_files"
typeset -i art_file_depth
art_file_depth=`echo $art_files | grep -oi "\/" | wc -l`
art_file_depth=$art_file_depth+1
echo "file_depth : $art_file_depth"
Art_File_Name=`echo ${art_files} | cut -d'/' -f${art_file_depth}`
echo "File Name : $Art_File_Name"
Destination_File=${Destination_Directory}'/'${Art_File_Name}
echo "will copy $Art_File_Name from $art_files to $Destination_File"
cp "$art_files" "$Destination_File"
done
else
echo "did nothing!!!"
fi
done
done
IFS=$oIFS
feel free to change, improve, distribute.
Caglar
Try this out:
#!/bin/bash
src_dir="in"
dst_dir="out"
find ${src_dir} -type f -print0|while IFS= read -r -d '' src_file; do
dst_file=${src_file/$src_dir/$dst_dir}
echo "src_file=${src_file} dst_file=${dst_file}"
mkdir -pv `dirname $dst_file`
# use above variables and run convert command with it here
done
To test how it works:
mkdir in out
cd in
mkdir 1 2 3
find . -type d -exec touch {}/foo {}/bar {}/baz \;
cd ..
./run_my_script.sh
Now you only need to attach your convert function/script/command/whatever and improve it to read src_dir and dst_dir from the command line (I would recommend man bash - > getopts)

bash echo string >> file does not work

I wrote the following script:
for filename in `find . -name '*'.cpp | grep $IN_REGEX | grep -v $OUT_REGEX`
do
echo "Output file is $OUTPUT_FILE"
count=`git log --pretty=format: --name-only $filename | grep -v ^$ | wc -l`
echo "$count $filename" >> $OUTPUT_FILE
done
But nothing gets written into the output file.
Please note:
I have set the values for OUTPUT_FILE, IN_REGEX and OUT_REGEX.
The code inside the loop is being executed. I checked this with an sh -x invokation.
When I remove the >> $OUTPUT_FILE I get the output.
I tried a touch $OUTPUT_FILE inside the script and that is working fine.
Can someone please point out what is my mistake here?
This line of code
for filename in `find . -name '*'.cpp
is a
(break on space in file names)
You should instead do :
while IFS= read -r file; do
echo "Output file is $OUTPUT_FILE"
count=$(git log --pretty=format: --name-only "$file" | grep -v '^$' | wc -l)
echo "$count $file" >> "$OUTPUT_FILE"
done < <(find . -name '*.cpp' | grep "$IN_REGEX" | grep -v "$OUT_REGEX")
For this to work, ensure that $OUTPUT_FILE have a path in it.

Resources