I'm writing a script to check if there actually is a directory that has content and a normal size, and see if there is a directory older then 36 hours, if not it should alert me.
However I'm having trouble using the directories as variable.
When I execute the script it returns: ./test.sh: line 5: 1: No such file or directory.
I tried ALLDIR=$(ls /home/customers/*/ as well but returned the same error.
What am I doing wrong? Below is the script.
Thanks a lot in advance!!
#!/bin/bash
ALLDIR=$(find * /home/customers/*/ -maxdepth 2 -mindepth 2)
for DIR in ${ALLDIR}
do
if [[ $(find "$DIR" -maxdepth 1 -type d -name '*' ! -mtime -36 | wc -l = <1 ) ]]; then
mail -s "No back-ups found today at $DIR! Please check the issue!" test#example.com
exit 1
fi
done
for DIR in ${ALLDIR}
do
if [[ $(find "$DIR" -mindepth 1 -maxdepth 1 -type d -exec du -ks {} + | awk '$1 <= 50' | cut -f 2- ) ]]; then
mail -s "Backup directory size is too small for $DIR, please check the issue!" test#example.com
exit 1
fi
done
For a start, to loop through all directories a fixed level deep, use this:
for dir in /home/customers/*/*/*/
A pattern ending in a slash / will only match directories.
Note that $dir is a lowercase variable name, don't use uppercase ones as they may clash with shell internal/environment variables.
Next, your conditions are a bit broken - you don't need to use a [[ test here:
if ! find "$dir" -maxdepth 1 -type d ! -mtime -36 | grep -q .
If anything is found, find will print it and grep will quietly match anything, so the pipeline will exit successfully. The ! at the start negates the condition, so the if branch will only be taken when this doesn't happen, i.e. when nothing is found. -name '*' is redundant.
You can do something similar with the second if, removing the [[ and $() and using grep -q . to test for any output. I guess the cut part is redundant too.
Related
I can get the amount of dirs with command
ls -dtr */ | wc -l
But how do I specifically delete the N-1 directories, leaving the most recent one?
As usual for any bash operation on "a bunch of files", you have to be aware of the pain that is spaces and newlines, which may legally appear in file names. These break naïve xargs / for based approaches and require some extra hoops to jump.
Many tools support -z or -0 options, which use NUL bytes as line separator instead of newlines -- NUL may never be part of a file name.
Unfortunately, ls is not one of them, so we have to go through find to get the latest directory.
find . -maxdepth 1 -mindepth 1 -type d
This gets you all the directories in the current directory. (Note: As opposed to your ls -dtr */, this will also find "hidden" directories, i.e. ones starting with a .. Add ! -name ".*" to avoid that.)
-maxdepth 1 avoids recursion, -mindepth 1 keeps the parent directory (.) out of the list.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0"
This lists the directories and their timestamps, using NUL instead of newline for line separation. (NUL can never be part of a file name.)
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z
This sorts the results, using NUL instead of newline for line separation.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z | head -z -n -1
This takes everything but the last entry (the latest directory and its timestamp) from the list, using NUL instead of newline for line separation.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z | head -z -n -1 | cut -z -d' ' -f 2-
Using NUL instead of newline for line separation and space as field delimiter, this filters the first field (the timestamp) from the output.
find . -maxdepth 1 -mindepth 1 -type d -printf "%T+ %f\0" | sort -z | head -z -n -1 | cut -z -d' ' -f 2- | xargs -0 rm -rf
Using NUL instead of newline for line separation, this calls rm -rf for each entry.
Use tail:
ls -dt */ | tail -n +2 | while read dirName; do rmdir "$dirName"; done
When specifying the -n option of tail with a +, it skips the first lines.
This will list the directories sorted by date, skip the most recent one and keep the rest, which will be deleted. You can replace rmdir with rm -rf following your use case.
You can try it out:
mkdir test && cd test
for i in {1..100}; do mkdir subdir-$i; done
ls -dt */ | tail -n +2 | while read dirName; do rmdir "$dirName"; done
ls
Output:
subdir-100
EDIT: if any of the directory names contain newline characters, this approach will fail because of tail. See #DevSolar's answer for a more complete one in that case.
This is a pure Bash solution:
shopt -s nullglob # Globs that match nothing expand to nothing
shopt -s dotglob # Globs include files whose names start with '.'
newest_dir=
for sdir in */ ; do
dir=${sdir%/} # Remove slash to enable a symlink check
[[ -L $dir ]] && continue # Skip symlinks to directories
if [[ -z $newest_dir ]] ; then
newest_dir=$dir
elif [[ $dir -nt $newest_dir ]] ; then
echo rm -rf -- "$newest_dir"
newest_dir=$dir
else
echo rm -rf -- "$dir"
fi
done
In its current form it just prints the commands to remove the old subdirectories. Remove the echos to make it functional.
So many ways, here's another. :)
rm -rf `ls -1dt * | tail -n +2 | tr '\n' ' '`
First, the ls -1dt * just lists all the directories sorted by modification time, one per line. Then tail -n +2 removes the first, most recently modified. Then tr '\n ' ' puts them all into a single line. Then the rm -rf the dirs.
I am trying to write a one liner to find the number of files in each home directory. I am trying to do this as the other day I had situation where I ran out of inodes on /home. It took me a long time to find the offender and I want to shorten this process. This is what i have but it is not working.
for i in /home/*; do if [ -d "$i" ]; then cd $i find . -xdev -maxdepth 100 -type f |wc -l; fi done
When I run it, it prints a 0 for each home directory, and I remain in roots home directory.
However when I run just this part:
for i in /home/*; do if [ -d "$i" ]; then cd $i; fi done
I wind up in the last home directory leading me to believe I traversed them all.
And when I run this in each users home directory:
find . -xdev -maxdepth 100 -type f |wc -l
I get a legit answer.
You're missing a terminating character after your cd. But more importantly, using cd can cause unwanted errors if you're not careful, try below instead (cd not needed).
for i in /home/*; do [ -d "$i" ] && echo "$i" && find "$i" -xdev -maxdepth 100 -type f | wc -l; done
Since find can take multiple paths, you don't need a loop:
find /home/*/ -xdev -maxdepth 100 -type f | wc -l
To avoid any issues with filenames containing newlines (rare, yes), you can take advantage of an additional GNU extension to find (you're using -maxdepth, so I assume you can use -printf as well):
find /home/*/ -xdev -maxdepth 100 -type f -printf "." | wc -c
Since you aren't actually using the name of the file for counting, replace it with a single-character string, then count the length of the resulting string.
I used this command to find all the directories containing .mp3 in the current directory, and filtered out only the directory names:
find . -iname "*.mp3" | sed -e 's!/[^/]*$!!' -e 's!^\./!!' | sort -u
I now want the opposite, but I found it a little harder. I can't just add a '!' to the find command since it'll only exclude .mp3 when printing them not find directories that do not contain .mp3.
I googled this and searched on stackoverflow and unix.stackexchange.com.
I have tried this script so far and it returns this error:
#!/bin/bash
find . -type d | while read dir
do
if [[! -f $dir/*.mp3 ]]
then
echo $dir
fi
done
/home/user/bin/try.sh: line 5: [[!: command not found
#!/bin/bash
find . -type d | while read dir
do
if [! -f $dir/*.mp3 ]
then
echo $dir
fi
done
/home/user/bin/try.sh: line 5: [!: command not found
#!/bin/bash
find . -type d | while read dir
do
if [[! -f "$dir/*.mp3" ]]
then
echo $dir
fi
done
/home/user/bin/try.sh: line 5: [!: command not found
I'm thinking it has to do with multiple arguments for the test command.
Since I'm testing all the directories the variable is going to change, and I use a wildcard for the filenames.
Any help is much appreciated. Thank You.
[ "$(echo $dir/*.mp3)" = "$dir/*.mp3" ]
should work.
Or simply add a space between '[' and '!'
A method that is probably significantly faster is
if find "$dir" -name '*.mp3' -quit ; then
: # there are mp3-files in there.
else
; # no mp3:s
fi
Okay, I solved my own answer by using a counter.
I don't know how efficient it is, but it works. I know it can be made better. Please feel free to critique.
find . -type d | while read dir
do
count=`ls -1 "$dir"/*.mp3 2>/dev/null | wc -l`
if [ $count = 0 ]
then
echo $dir
fi
done
This prints all directories not containing MP3s It also shows sub-directories thanks to the find command printing directories recursively.
I ran a script to automatically download cover art for my mp3 collection. It put a file called "cover.jpg" in the directory for each album for which it could retrieve the art. I needed to check for which albums the script had failed - i.e. which CDs (directories) did not contain a file called cover.jpg. This was my effort:
find . -maxdepth 1 -mindepth 1 -type d | while read dir; do [[ ! -f $dir/cover.jpg ]] && echo "$dir has no cover art"; done
The maxdepth 1 stops the find command from descending into a hidden directory which my WD My Cloud NAS server had created for each album and placed a default generic disc image. (This got cleared during the next scan.)
Edit: cd to the MP3 directory and run it from there, or change the . in the command above to the path to point to it.
Using bash, how can one get the number of files in a folder, excluding directories from a shell script without the interpreter complaining?
With the help of a friend, I've tried
$files=$(find ../ -maxdepth 1 -type f | sort -n)
$num=$("ls -l" | "grep ^-" | "wc -l")
which returns from the command line:
../1-prefix_blended_fused.jpg: No such file or directory
ls -l : command not found
grep ^-: command not found
wc -l: command not found
respectively. These commands work on the command line, but NOT with a bash script.
Given a file filled with image files formatted like 1-pano.jpg, I want to grab all the images in the directory to get the largest numbered file to tack onto the next image being processed.
Why the discrepancy?
The quotes are causing the error messages.
To get a count of files in the directory:
shopt -s nullglob
numfiles=(*)
numfiles=${#numfiles[#]}
which creates an array and then replaces it with the count of its elements. This will include files and directories, but not dotfiles or . or .. or other dotted directories.
Use nullglob so an empty directory gives a count of 0 instead of 1.
You can instead use find -type f or you can count the directories and subtract:
# continuing from above
numdirs=(*/)
numdirs=${#numdirs[#]}
(( numfiles -= numdirs ))
Also see "How can I find the latest (newest, earliest, oldest) file in a directory?"
You can have as many spaces as you want inside an execution block. They often aid in readability. The only downside is that they make the file a little larger and may slow initial parsing (only) slightly. There are a few places that must have spaces (e.g. around [, [[, ], ]] and = in comparisons) and a few that must not (e.g. around = in an assignment.
ls -l | grep -v ^d | wc -l
One line.
How about:
count=$(find .. -maxdepth 1 -type f|wc -l)
echo $count
let count=count+1 # Increase by one, for the next file number
echo $count
Note that this solution is not efficient: it spawns sub shells for the find and wc commands, but it should work.
file_num=$(ls -1 --file-type | grep -v '/$' | wc -l)
this is a bit lightweight than a find command, and count all files of the current directory.
The most straightforward, reliable way I can think of is using the find command to create a reliably countable output.
Counting characters output of find with wc:
find . -maxdepth 1 -type f -printf '.' | wc --char
or string length of the find output:
a=$(find . -maxdepth 1 -type f -printf '.')
echo ${#a}
or using find output to populate an arithmetic expression:
echo $(($(find . -maxdepth 1 -type f -printf '+1')))
Simple efficient method:
#!/bin/bash
RES=$(find ${SOURCE} -type f | wc -l)
Get rid of the quotes. The shell is treating them like one file, so it's looking for "ls -l".
REmove the qoutes and you will be fine
Expanding on the accepted answer (by Dennis W): when I tried this approach I got incorrect counts for dirs without subdirs in Bash 4.4.5.
The issue is that by default nullglob is not set in Bash and numdirs=(*/) sets an 1 element array with the glob pattern */. Likewise I suspect numfiles=(*) would have 1 element for an empty folder.
Setting shopt -s nullglob to disable nullglobbing resolves the issue for me. For an excellent discussion on why nullglob is not set by default on Bash see the answer here: Why is nullglob not default?
Note: I would have commented on the answer directly but lack the reputation points.
Here's one way you could do it as a function. Note: you can pass this example, dirs for (directory count), files for files count or "all" for count of everything in a directory. Does not traverse tree as we aren't looking to do that.
function get_counts_dir() {
# -- handle inputs (e.g. get_counts_dir "files" /path/to/folder)
[[ -z "${1,,}" ]] && type="files" || type="${1,,}"
[[ -z "${2,,}" ]] && dir="$(pwd)" || dir="${2,,}"
shopt -s nullglob
PWD=$(pwd)
cd ${dir}
numfiles=(*)
numfiles=${#numfiles[#]}
numdirs=(*/)
numdirs=${#numdirs[#]}
# -- handle input types files/dirs/or both
result=0
case "${type,,}" in
"files")
result=$((( numfiles -= numdirs )))
;;
"dirs")
result=${numdirs}
;;
*) # -- returns all files/dirs
result=${numfiles}
;;
esac
cd ${PWD}
shopt -u nullglob
# -- return result --
[[ -z ${result} ]] && echo 0 || echo ${result}
}
Examples of using the function :
folder="/home"
get_counts_dir "files" "${folder}"
get_counts_dir "dirs" "${folder}"
get_counts_dir "both" "${folder}"
Will print something like :
2
4
6
Short and sweet method which also ignores symlinked directories.
count=$(ls -l | grep ^- | wc -l)
or if you have a target:
count=$(ls -l /path/to/target | grep ^- | wc -l)
I would like to find the newest sub directory in a directory and save the result to variable in bash.
Something like this:
ls -t /backups | head -1 > $BACKUPDIR
Can anyone help?
BACKUPDIR=$(ls -td /backups/*/ | head -1)
$(...) evaluates the statement in a subshell and returns the output.
There is a simple solution to this using only ls:
BACKUPDIR=$(ls -td /backups/*/ | head -1)
-t orders by time (latest first)
-d only lists items from this folder
*/ only lists directories
head -1 returns the first item
I didn't know about */ until I found Listing only directories using ls in bash: An examination.
This ia a pure Bash solution:
topdir=/backups
BACKUPDIR=
# Handle subdirectories beginning with '.', and empty $topdir
shopt -s dotglob nullglob
for file in "$topdir"/* ; do
[[ -L $file || ! -d $file ]] && continue
[[ -z $BACKUPDIR || $file -nt $BACKUPDIR ]] && BACKUPDIR=$file
done
printf 'BACKUPDIR=%q\n' "$BACKUPDIR"
It skips symlinks, including symlinks to directories, which may or may not be the right thing to do. It skips other non-directories. It handles directories whose names contain any characters, including newlines and leading dots.
Well, I think this solution is the most efficient:
path="/my/dir/structure/*"
backupdir=$(find $path -type d -prune | tail -n 1)
Explanation why this is a little better:
We do not need sub-shells (aside from the one for getting the result into the bash variable).
We do not need a useless -exec ls -d at the end of the find command, it already prints the directory listing.
We can easily alter this, e.g. to exclude certain patterns. For example, if you want the second newest directory, because backup files are first written to a tmp dir in the same path:
backupdir=$(find $path -type -d -prune -not -name "*temp_dir" | tail -n 1)
The above solution doesn't take into account things like files being written and removed from the directory resulting in the upper directory being returned instead of the newest subdirectory.
The other issue is that this solution assumes that the directory only contains other directories and not files being written.
Let's say I create a file called "test.txt" and then run this command again:
echo "test" > test.txt
ls -t /backups | head -1
test.txt
The result is test.txt showing up instead of the last modified directory.
The proposed solution "works" but only in the best case scenario.
Assuming you have a maximum of 1 directory depth, a better solution is to use:
find /backups/* -type d -prune -exec ls -d {} \; |tail -1
Just swap the "/backups/" portion for your actual path.
If you want to avoid showing an absolute path in a bash script, you could always use something like this:
LOCALPATH=/backups
DIRECTORY=$(cd $LOCALPATH; find * -type d -prune -exec ls -d {} \; |tail -1)
With GNU find you can get list of directories with modification timestamps, sort that list and output the newest:
find . -mindepth 1 -maxdepth 1 -type d -printf "%T#\t%p\0" | sort -z -n | cut -z -f2- | tail -z -n1
or newline separated
find . -mindepth 1 -maxdepth 1 -type d -printf "%T#\t%p\n" | sort -n | cut -f2- | tail -n1
With POSIX find (that does not have -printf) you may, if you have it, run stat to get file modification timestamp:
find . -mindepth 1 -maxdepth 1 -type d -exec stat -c '%Y %n' {} \; | sort -n | cut -d' ' -f2- | tail -n1
Without stat a pure shell solution may be used by replacing [[ bash extension with [ as in this answer.
Your "something like this" was almost a hit:
BACKUPDIR=$(ls -t ./backups | head -1)
Combining what you wrote with what I have learned solved my problem too. Thank you for rising this question.
Note: I run the line above from GitBash within Windows environment in file called ./something.bash.