Sorting loop function creates infinite subdirectories - bash

I routinely have to sort large amounts of images in multiple folders into two 2 file types, ".JPG" and ".CR2". I'm fairly new to bash but have created a basic script that will sort through one individual folder successfully and divide these file types into distinct folders.
I'm having problems scaling this up to automatically loop through subdirectories. My current script creates an infinite loop of new subfolders until terminal times out.
How can I use the loop function without having it cycle through new folders?
function f {
cd "$1"
count=`ls -1 *.JPG 2>/dev/null | wc -l`
if [ $count != 0 ]; then
echo true
mkdir JPG; mv *.JPG jpg
else
echo false
fi
count=`ls -1 *.CR2 2>/dev/null | wc -l`
if [ $count != 0 ]; then
echo true
mkdir RAW; mv *.CR2 raw;
else
echo false
fi
for d in * .[!.]* ..?*; do
cd "$1"
test -d "$1/$d" && f "$1/$d"
done
}
f "`pwd`"

I still advise people to use find instead of globulation * in scripts. The * may not work reliably always, may fail and confuse.
First we create directories to move to:
mkdir -p ./jpg ./cr2
Note that -p in mkdir will make mkdir not fail in case the directory already exists.
Use find. Find all files named *.JPG and move each file to jpg :
find . -maxdepth 1 -mindepth 1 -name '*.JPG' -exec mv {} ./jpg \;
// similar
find . -maxdepth 1 -mindepth 1 -name '*.CR2' -exec mv {} ./cr2 \;
The -maxdepth 1 -mindepth 1 is so that the find does not scan the directory recursively, which is the default. You can remove them, but If you want, you can add -type f to include files only.
Notes to your script:
Don't parse ls output
You can use the find . -mindepth 1 -maxdepth 1 -file '*.jpg' -print . | wc -c to get the number of files in a directory instead.
for d in * .[!.]* ..?*; do I have a vague idea what this is supposed to do, some kind of recursively scanning the directory. Buf if the directory JPG is inside $(pwd) then you will scan infinitely into yourself and move the file into yourself etc... If the destination folder is outside current directory, just modify the find scripts by removing -mindepth 1, it will scan recursively then.
Don't use backticks, they are less readable and are deprecated. Use $( .. ) instead.

Related

How to get file count and names in directory on bash

I want to get the file count & file names & folder names in directory:
mkdir -p /tmp/test/folder1
mkdir -p /tmp/test/folder2
touch /tmp/test/file1
touch /tmp/test/file2
file_names=$(find "/tmp/test" -mindepth 1 -maxdepth 1 -type f -print0 | xargs -0 -I {} basename "{}")
echo $file_names
here is the output:
file2 file1
For folder:
folder_names=$(find "/tmp/test" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -I {} basename "{}")
echo $folder_names
here is the output:
folder2 folder1
For count:
file_count=0 && $(find "/tmp/test" -mindepth 1 -maxdepth 1 -type f -print0 | let "file_count=file_count+1")
echo $file_count
folder_count=0 && $(find "/tmp/test" -mindepth 1 -maxdepth 1 -type d -print0 | let "folder_count=folder_count+1")
echo $folder_count
The file_count and folder_count does not work
Question 1:
How to get the correct file_count and folder_count?
Question 2:
Is it possible for getting names into an array and check the count from array size?
The answer to the second question is really the answer to the first, too.
mapfile -d '' files < <( find /tmp/test -type f \
-mindepth 1 -maxdepth 1 \
-printf '%f\0')
echo "${#files} files"
printf '%s\n' "${files[#]}"
The use of double quotes and # in the array expansion are essential for printing file names with whitespace correctly. The use of a null byte terminator between file names ensures that even newlines in file names are disambiguated.
Notice also the use of -printf with a specific format string to avoid having to run basename separately. However, the -printf option and its various format strings, as well as the -print0 option you used, are a GNU find extension, and thus not portable. (Linux typically ships with GNU tools; on other platforms, they are obviously easy to install, but typically not installed out of the box.)
If you have an older version of Bash which doesn't support mapfiles, try an explicit loop:
files=()
while IFS= read -r -d $'\0' file; do
files+=("$file")
done < <(find ...)
If you don't have GNU find, a common workaround is to print a fixed string for each found file, and then the line or character count reliably reflects the number of found files.
find /tmp/test -type f \
-mindepth 1 -maxdepth 1 \
-exec printf . \; |
wc -c
Though then, how do you collect the file names? If (as in your case) you don't require recursion into subdirectories, simply loop over all items in the directory.
In which case, again, the number of items in the collected array will also tell you how many there are.
files=()
dirs=()
for item in /tmp/test/*; do
if [[ -f "$item"]]; then
files+=("$item")
elif [[ -d "$item" ]]; then
dirs+=("$item")
fi
done
echo "${#dirs[#] directories}
printf '- %s\n' "${dirs[#]}"
echo "${#files[#]} files"
printf '%s\n' "${dirs[#]}"
For a further discussion, see also https://mywiki.wooledge.org/BashFAQ/020
Needlessly collecting items into an array so you can loop over it once is a common beginner antipattern. If you just want to process each item in isolation and then forget it, there is no need to waste memory on remembering all the other items - just loop over them directly.
As an aside, running find in a subprocess will create a new shell instance with its own set of variables; thus in your attempt, the pipe to let would increment from 0 to 1 each time you ran it (though of course, piping to let also does not do anything useful anyway).

Move files from several folders to subfolders

I have many folders with files in them, in each of them I have a subfolder called wvfm . What I want to do is move all the files into each of the wvfm folder.
I tried doing the line below but it is not working
for i in "./*"; mv $i "./wvfm"; done
but that didn't work quite right
Use a for loop (for i in */; do) to list the files, then move to the folders to list and save all files in to a variable (F=$(ls)).
Then move all your files with excluding your folder (mv ${F/wvfm/} wvfm), like this:
#!/bin/bash
subdirectory="wvfm"
for i in */; do
cd $i
F=$(ls)
mv ${F/$subdirectory/} $subdirectory
cd ../
done
find * -maxdepth 0 -type d | xargs -n 1 -I {} mv {}/* {}/wvfm
should do the trick; quick and dirty. Not a MacOS user, but works in bash.
Explanation:
find * -maxdepth 0 -type d find all directories at depth 0 (ie: do not descend dirs),
pipe to xargs, using options -n 1, operate on one value at a time, -I replace-str, string replacement (see man xargs).
action command: mv {}/* {}/wvfm substitues to mv dirA/* dirA/wvfm for each dir match.
You will get an "error", mv: cannot move /wvfm' to a subdirectory of itself, 'wvfm/wvfm', but you can ignore / take advantage of it (quick and dirty).
You could cover all bases with:
for entry in $(find * -maxdepth 0 -type d); do
(cd $entry; [[ -d wvfm ]] && \
find * -maxdepth 0 ! -name wvfm -exec mv {} wvfm \;
)
done
find * -maxdepth 0 -type d, again, find only the top-level directories,
in a subshell, change to the input directory, if there's a directory wvfm,
look for all contents except the wvfm directory and -exec the mv command
exiting the subshell leaves you back in the starting (root) directory, ready for next input.

Bash: List directories with a type of file, but missing another type of file

I'm new(ish) to using Bash and I'm trying to figure out how to combine a few different things into one script.
I'm looking for file transfers that were interrupted. These folders contain image files (either jpgs or pngs), but are missing another specific file (finished.txt).
Here is what I'm using to find folders with images (from here):
for f in */incoming/ ; do
log_f="${f//\//}"
echo "searching $f"
find "$f" -iname "*jpg*" -o -iname "*png*" > "/output/${log_f}.txt"
echo "$f finished"
done
Then, I'm running this command to find folders that are missing the finished.txt file (from here):
find -mindepth 2 -maxdepth 2 -type d '!' -exec test -e "{}/finished.txt" ';' -print
Is there a way to combine them so I have a list of folders which have jpg or png files, but don't have finished.txt? Also, If I want to add -mtime, where do I put that?
Alternatively, if there's a better/faster way to do this, I'm interested in that too.
Thanks!
From the first pass when you get the files with jpg/png you can get the directory by using dirname. The list of directories can then be used for iterating over and looking for finished.txt file. On finding you can skip the directory if not print it out.
Something as below should do the needful
for i in `find "$f" -iname "*jpg*" -o -iname "*png*" -exec dirname {} \;`
do
ls $i | grep finished >/dev/null
if [ $? -eq 1 ]; then
echo $i
fi
done
Add " | sort | uniq" at the end of find command to perhaps remove the duplicates. Something like
find "$f" -iname "jpg" -o -iname "png" -exec dirname {} \; | sort | uniq

why is part of my one liner not being executed

I am trying to write a one liner to find the number of files in each home directory. I am trying to do this as the other day I had situation where I ran out of inodes on /home. It took me a long time to find the offender and I want to shorten this process. This is what i have but it is not working.
for i in /home/*; do if [ -d "$i" ]; then cd $i find . -xdev -maxdepth 100 -type f |wc -l; fi done
When I run it, it prints a 0 for each home directory, and I remain in roots home directory.
However when I run just this part:
for i in /home/*; do if [ -d "$i" ]; then cd $i; fi done
I wind up in the last home directory leading me to believe I traversed them all.
And when I run this in each users home directory:
find . -xdev -maxdepth 100 -type f |wc -l
I get a legit answer.
You're missing a terminating character after your cd. But more importantly, using cd can cause unwanted errors if you're not careful, try below instead (cd not needed).
for i in /home/*; do [ -d "$i" ] && echo "$i" && find "$i" -xdev -maxdepth 100 -type f | wc -l; done
Since find can take multiple paths, you don't need a loop:
find /home/*/ -xdev -maxdepth 100 -type f | wc -l
To avoid any issues with filenames containing newlines (rare, yes), you can take advantage of an additional GNU extension to find (you're using -maxdepth, so I assume you can use -printf as well):
find /home/*/ -xdev -maxdepth 100 -type f -printf "." | wc -c
Since you aren't actually using the name of the file for counting, replace it with a single-character string, then count the length of the resulting string.

Finding empty directories

I need to find empty directories for a given list of directories.
Some directories have directories inside it.
If inside directories are also empty I can say main directory is empty otherwise it's not empty.
How can I test this?
For example:
A>A1(file1),A2 this is not empty beacuse of file1
B>B1(no file) this is empty
C>C1,C2 this is empty
It depends a little on what you want to do with the empty directories. I use the command below when I wish to delete all empty directories within a tree, say test directory.
find test -depth -empty -delete
One thing to notice about the command above is that it will also remove empty files, so use the -type d option to avoid that.
find test -depth -type d -empty -delete
Drop -delete to see the files and directories matched.
If your definition of an empty directory tree is that it contains no files then you be able to stick something together based on whether find test -type f returns anything.
find is a great utility, and RTFM early and often to really understand how much it can do :-)
You can use the following command:
find . -type d -empty
Check whether find <dir> -type f outputs anything. Here's an example:
for dir in A B C; do
[ -z "`find $dir -type f`" ] && echo "$dir is empty"
done
find directory -mindepth 1 -type d -empty -delete
This is the version that I found most interesting. If executed from inside directory, it will delete all empty directories below (a directory is considered empty if it only contains empty directories).
The mindepth option prevents the directory itself from being deleted if it happens to be empty.
find . -type d -empty
finds and lists empty directories and sub-directories in the current tree.
E.g. resulting list of empty dirs and subdirs:
./2047
./2032
./2049
./2063
./NRCP26LUCcct1/2039
./NRCP26LUCcct1/2054
./NRCP26LUCcct1/2075
./NRCP26LUCcct1/2070
No operation is made on the directories. They are simply listed.
This works for me.
Just find empty dirs
In order to just find empty directories (as specified in the question title), the mosg's answer is correct:
find -type d -empty
But -empty may not be available on very old find versions (this is the case of HP-UX for example). If this is your case, see the techniques described in below section Is a directory empty?.
Delete empty dirs
This is a bit tricky: Suppose a directory MyDir contains empty directories. After removing these empty directories, MyDir will become an empty directory and should also be removed. Therefore I use the command rmdir with the option --parents (or -p) that also removes parent directories when possible:
find -type d -empty -exec rmdir -vp --ignore-fail-on-non-empty {} +
On older find version the statement + is not yet supported, therefore you may use ; instead:
find -type d -empty -exec rmdir -vp --ignore-fail-on-non-empty {} `;`
Is a directory empty?
Most of these answers explain how to check if a directory is empty. Therefore I provide here the three different techniques I know:
[ $(find your/dir -prune -empty) = your/dir ]
d=your/dir
if [ x$(find "$d" -prune -empty) = x"$d" ]
then
echo "empty (directory or file)"
else
echo "contains files (or does not exist)"
fi
a variation:
d=your/dir
if [ x$(find "$d" -prune -empty -type d) = x"$d" ]
then
echo "empty directory"
else
echo "contains files (or does not exist or is not a directory)"
fi
Explanation:
find -prune is similar than find -maxdepth 0 using less characters
find -type d prints directories only
find -empty prints the empty directories and files
> mkdir -v empty1 empty2 not_empty
mkdir: created directory 'empty1'
mkdir: created directory 'empty2'
mkdir: created directory 'not_empty'
> touch not_empty/file
> find empty1 empty2 not_empty -prune -empty
empty1
empty2
(( ${#files} ))
This trick is 100% bash but invokes (spawns) a sub-shell. The idea is from Bruno De Fraine and improved by teambob's comment. I advice this one if you use bash and if your script does not have to be portable.
files=$(shopt -s nullglob dotglob; echo your/dir/*)
if (( ${#files} ))
then
echo "contains files"
else
echo "empty (or does not exist or is a file)"
fi
Note: no difference between an empty directory and a non-existing one (and even when the provided path is a file).
[ $(ls -A your/dir) ]
This trick is inspired from nixCraft's article posted in 2007. Andrew Taylor answered in 2008 and gr8can8dian in 2011.
if [ "$(ls -A your/dir)" ]
then
echo "contains files"
else
echo "empty (or does not exist or is a file)"
fi
or the one-line bashism version:
[[ "$(ls -A your/dir)" ]] && echo "contains files" || echo "empty or ..."
Note: ls returns $?=2 when the directory does not exist. But no difference between a file and an empty directory.
How about rmdir *? That command will fail on non-empty directories.
This recursive function would seem to do the trick:
# Bash
findempty() {
find ${1:-.} -mindepth 1 -maxdepth 1 -type d | while read -r dir
do
if [[ -z "$(find "$dir" -mindepth 1 -type f)" ]] >/dev/null
then
findempty "$dir"
echo "$dir"
fi
done
}
Given this example directory structure:
.
|-- dir1/
|-- dir2/
| `-- dirB/
|-- dir3/
| `-- dirC/
| `-- file5
|-- dir4/
| |-- dirD/
| `-- file4
`-- dir5/
`-- dirE/
`-- dir_V/
The result of running that function would be:
./dir1
./dir5/dirE/dir_V
./dir5/dirE
./dir5
./dir2/dirB
./dir2
which misses /dir4/dirD. If you move the recursive call findempty "$dir" after the fi, the function will include that directory in its results.
The following command returns 1 if a directory is empty (or does not exists) and 0 otherwise (so it is possible to invert the return code with ! in a shell script):
find $dir -type d -prune -empty -exec false {} +
I created a simple structure as follows:
test/
test/test2/
test/test2/test2.2/
test/test3/
test/test3/file
The test/test3/file contains some junk text.
Issuing find test -empty returns "test/test2/test2.2" as the only empty directory.
a simple approach would be,
$ [ "$(ls -A /path/to/direcory)" ] && echo "not empty" || echo "its empty"
also,
if [ "$(ls -A /path/to/direcory)" ]; then
echo "its not empty"
else
echo "empty directory"
find . -name -type d -ls |awk '($2==0){print $11}'

Resources