ls command and size of files in shell script - bash

count=0; #count for counting
IFS='
'
for x in `ls -l $input`; #for loop using ls command
do
a=$(ls -ls | awk '{print $6}') #print[6] is sizes of file
echo $a
b=`echo $a | awk '{split($0,numbers," "); print numbers[1]}'`
echo $b
if [ $b -eq 0 ] # b is only size of a file
then
count=`expr $count + 1` #if b is zero , the count will increase one by one
fi
echo $count
done
I want to find 0 size files . I do that using find command. The second thing is I want to count number of has 0 size of files using ls command and awk. But It doesn't true code . What is my mistake ?

The -s test is true if a file has non-zero size. If that test fails for file, increment your empty-file count.
empty_files=0
for f in "$input"/*; do
[ -s "$f" ] || : $(( empty_files++ ))
done

Your main mistake is that you're parsing ls!
If you want to find (regular) files that are empty, and if you have a version of find that supports the -empty predicate, use it:
find . -type f -empty
Note that this will recurse in subfolders too; if you don't want that, use:
find . -maxdepth 1 -type f -empty
(assuming that your find also supports -maxdepth).
If you only want to count how many empty (regular) files you have:
find . -maxdepth 1 -type f -empty -printf x | wc -m
and if you want to perform both operations at the same time, i.e., print out the name or save them in an array for future use, and count them:
empty_files=()
while IFS= read -r -d '' f; do
empty_files+=( "$f" )
done < <(find . -maxdepth 1 -type f -empty -print0)
printf 'There are %d empty files:\n' "${#empty_files[#]}"
printf ' %s\n' "${empty_files[#]}"
With Bash≥4.4, you could use mapfile instead of the while-read loop:
mapfile -t -d '' empty_files < <(find . -maxdepth 1 -type f -empty -print0)
printf 'There are %d empty files:\n' "${#empty_files[#]}"
printf ' %s\n' "${empty_files[#]}"
For a POSIX-compliant way, use test with the -s option:
find . -type f \! -exec test -s {} \; -print
and if you don't want to recurse into subdirectories, you'll have to -prune them:
find . \! -name . -prune -type f \! -exec test -s {} \; -print
and if you want to count them:
find . \! -name . -prune -type f \! -exec test -s {} \; -exec printf x | wc -m
and here, if you want to perform both operations (count them and save them in an array for later use), use the previous while-read loop (or mapfile if you live in the future) with this find:
find . \! -name . -prune -type f \! -exec test -s {} \; -exec printf '%s\0' {} \;
Also see chepner's answer for a pure shell solution (needs minor tweaking to be POSIX compliant).
Regarding your comment
I want to count and delete [empty files]. How can I do that at the same time?
If you have GNU find (or a find that supports all the goodies):
find . -maxdepth 1 -type f -empty -printf x -delete | wc -m
if not,
find . \! -name . -prune -type f \! -exec test -s {} \; -printf x -exec rm {} \; | wc -m
Make sure that the -delete (or -exec rm {} \;) predicate is at the end! do not exchange the order of the predicates!

Related

Bash find or xargs evaluates variables and subshells only once

I noticed that find ... -exec ... {} \; or xargs -i ... {} seems to evaluate variables or subshells (like $RANDOM or $(uuidgen)) only once, even the command was executed mutiple times.
For example:
$ find . -type f -name \*.txt -exec echo "$RANDOM {}" \;
28855 ./foo/bar.txt
28855 ./foo/bar1.txt
28855 ./foo/bar2.txt
28855 ./foo/bar3.txt
28855 ./foo/bar4.txt
$ grep -lr SOME_TEXT --include=\*.txt | xargs -i echo "$RANDOM {}"
6153 ./foo/bar.txt
6153 ./foo/bar1.txt
6153 ./foo/bar2.txt
6153 ./foo/bar3.txt
6153 ./foo/bar4.txt
Is there a way to get a result like below?
1543 ./foo/bar.txt
543 ./foo/bar1.txt
57224 ./foo/bar2.txt
3525 ./foo/bar3.txt
18952 ./foo/bar4.txt
Yes. The variable expansion is performed after the line has been accepted, but before it has been executed. This means that the command that ends up being executed is
'/usr/bin/find' '.' '-type' 'f' '-name' '*.txt' '-exec' 'echo' '28855 {}' ';'
Two basic ways around this:
Use another bash that will delay the execution:
find . -type f -name \*.txt -exec bash -c 'echo "$RANDOM {}"' \;
Use a loop:
for file in $(find . -type f -name \*.txt -print)
do
echo "$RANDOM $file"
done
If your files have spaces, you have to do something different to preserve them:
mapfile -d '' files < <(find . -type f -name \*.txt -print0)
for file in "${files[#]}"
do
echo "$RANDOM $file"
done

find printf what is correct format in crontab?

i want to save a list of files and i want to do in cron but i dont know how to convert this command
echo $(find /tmp -type f -printf "%p||%s||||||") > /share/Public/serwer/test33.data
in terminal this command works ok but in cron is smothing wrong, file is empty, why?
i tried add slashes before variables %s %p
i tried many many other combinations:
echo $(find /tmp -type f -printf "%p||%s||||||") > /share/Public/serwer/test33.data
echo $(find /tmp -type f -printf "%p||%s||||||") > /share/Public/serwer/test32.data
echo $(find /tmp -type f) > $TMP_DIR/test31.data
echo $(find $BACKUP_DIR -type f -printf \%p) > $TMP_DIR/test30.data
echo $(find $BACKUP_DIR -type f -printf %p) > $TMP_DIR/test28.data
echo $(find $BACKUP_DIR -type f) > $TMP_DIR/test27.data
echo $(find $BACKUP_DIR -type f -printf "ab") > $TMP_DIR/test26.data
echo $(find $BACKUP_DIR -type f -printf "||") > $TMP_DIR/test25.data
echo $(find $BACKUP_DIR -type f -printf "%p||%s\r\n") > $TMP_DIR/test01.data
echo $(find $BACKUP_DIR -type f -printf "\%p||\%s\r\n") > $TMP_DIR/test02.data
echo $(find $BACKUP_DIR -type f -printf "\%p") > $TMP_DIR/test03.data
echo $(find $BACKUP_DIR -type f -printf "\%s") > $TMP_DIR/test04.data
echo $(find $BACKUP_DIR -type f -printf "\\%s") > $TMP_DIR/test05.data
echo $(find $BACKUP_DIR -type f -printf "\\%p") > $TMP_DIR/test06.data
echo $(find $BACKUP_DIR -type f -printf "\r\n") > $TMP_DIR/test07.data
echo $(find $BACKUP_DIR -type f -printf) > $TMP_DIR/test10.data
echo $(find $BACKUP_DIR -type f -printf) > $TMP_DIR/test11.data
echo $(find $BACKUP_DIR -type f -printf "%p") > $TMP_DIR/test12.data
echo $(find $BACKUP_DIR -type f -printf "\%p") > $TMP_DIR/test13.data
echo `find $BACKUP_DIR -type f -printf "%p"` > $TMP_DIR/test14.data
echo `find $BACKUP_DIR -type f -printf "\\\\%p"` > $TMP_DIR/test15.data
echo `find $BACKUP_DIR -type f -printf \%p` > $TMP_DIR/test16.data
echo `find $BACKUP_DIR -type f -printf '\%p'` > $TMP_DIR/test17.data
echo `find $BACKUP_DIR -type f -printf '\\%p'` > $TMP_DIR/test18.data
echo `find $BACKUP_DIR -type f -printf '\\\%p'` > $TMP_DIR/test19.data
echo `find $BACKUP_DIR -type f -printf '\\\\%p'` > $TMP_DIR/test20.data
echo `find $BACKUP_DIR -type f -printf \%'s` > $TMP_DIR/test21.data
echo `find $BACKUP_DIR -type f -printf \%'p` > $TMP_DIR/test22.data
echo `find $BACKUP_DIR -type f -printf \%'p'` > $TMP_DIR/test23.data
echo `find $BACKUP_DIR -type f -printf \%'s'` > $TMP_DIR/test24.data
nothing work
This crontab entry should work :
* * * * * find /tmp -type f -printf "\%p||\%s||||||\n" > /share/Public/serwer/test33.data 2>/tmp/crontab.err
i added full path to command find, now it works correctly
/share/CACHEDEV1_DATA/.qpkg/Qapache/bin/find $BACKUP_DIR -type f -printf "%p||%s\r"
thanks for help
Thanks Philippe i do that and i have smothing like this:
find: unrecognized: -printf
BusyBox v1.24.1 (2021-09-23 02:31:15 CST) multi-call binary.
Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]
Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'
-L,-follow Follow symlinks
-H ...on command line only
-xdev Don't descend directories on other filesystems
-maxdepth N Descend at most N levels. -maxdepth 0 applies
actions to command line arguments only
-mindepth N Don't act on first N levels
-depth Act on directory *after* traversing it
Actions:
( ACTIONS ) Group actions for -o / -a
! ACT Invert ACT's success/failure
ACT1 [-a] ACT2 If ACT1 fails, stop, else do ACT2
ACT1 -o ACT2 If ACT1 succeeds, stop, else do ACT2
Note: -a has higher priority than -o
-name PATTERN Match file name (w/o directory name) to PATTERN
-iname PATTERN Case insensitive -name
-path PATTERN Match path to PATTERN
-ipath PATTERN Case insensitive -path
-regex PATTERN Match path to regex PATTERN
-type X File type is X (one of: f,d,l,b,c,...)
-perm MASK At least one mask bit (+MASK), all bits (-MASK),
or exactly MASK bits are set in file's mode
-mtime DAYS mtime is greater than (+N), less than (-N),
or exactly N days in the past
-mmin MINS mtime is greater than (+N), less than (-N),
or exactly N minutes in the past
-newer FILE mtime is more recent than FILE's
-inum N File has inode number N
-user NAME/ID File is owned by given user
-group NAME/ID File is owned by given group
-size N[bck] File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
+/-N: file size is bigger/smaller than N
-links N Number of links is greater than (+N), less than (-N),
or exactly N
-prune If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
-print Print file name
-print0 Print file name, NUL terminated
-exec CMD ARG ; Run CMD with all instances of {} replaced by
file name. Fails if CMD exits with nonzero
-exec CMD ARG + Run CMD with {} replaced by list of file names
-delete Delete current file/directory. Turns on -depth option

Using command substitution in find -exec

How can I use command substitution in find … -exec … to avoid using xargs in the following command?
find -L -- /path/to/directory -mindepth 2 -maxdepth 2 -type d -exec dirname '{}' \; | xargs basename -a
I tried the following using command substitution, but it output . for each result instead of the desired output:
find -L -- /path/to/directory -mindepth 2 -maxdepth 2 -type d -exec basename "$(dirname '{}')" \;
Your first command will return strange results if a path contains whitespace.
Use a small shell script:
find -L -- . -mindepth 2 -maxdepth 2 -type d -exec sh -c 'basename "$(dirname "{}")"' \;
Alternative syntax to pass one path argument to the script:
find -L -- . -mindepth 2 -maxdepth 2 -type d -exec sh -c 'basename "$(dirname "$1")"' sh {} \;
Or pass as many arguments to the script as possible:
find -L -- . -mindepth 2 -maxdepth 2 -type d -exec sh -c '
for path do
basename "$(dirname "$path")"
done
' sh {} +
With GNU utilities it's possible to output NUL-terminated strings with dirname passed to xargs -0. The basename command is not run if there are no arguments (-r):
find -L -- . -mindepth 2 -maxdepth 2 -type d -exec dirname -z {} + | xargs -r0 basename -a

How can I properly execute a command with lists using `sh -c`?

I have this command to recursively find directories that contain mustExist.js but not cannotExist.js:
comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u)
It works fine.
Now, I must pass it as a string to a node.js automation script. The script picks up the string and runs it as sh -c <string>. I cannot change that part.
So I pass this string:
'comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \\; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \\; | sort -u)'
However, I always encounter this error:
Warning: Command failed: /bin/sh -c comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u)
/bin/sh: 1: Syntax error: "(" unexpected
When I pass this string:
'"comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \\; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \\; | sort -u)"'
I get:
Warning: Command failed: /bin/sh -c "comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u)"
/bin/sh: 1: comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u): not found
When I manually try using sh to immitate the automation script:
sh -c comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u)
I get a different error:
comm: missing operand
Or with quotes:
sh -c "comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u)"
I get:
/bin/sh: 1: Syntax error: "(" unexpected
Other quotes:
sh -c 'comm -13 <(find . -type f -name cannotExist.js -exec dirname {} \; | sort -u) <(find . -type f -name mustExist.js -exec dirname {} \; | sort -u)'
/bin/sh: 1: Syntax error: "(" unexpected
Is it possible to do this somehow?
There is a much simpler and more efficient command you can use:
find . -type d -exec sh -c 'test -e "$1"/mustExist.js && ! test -e "$1"/cannotExist.js' _ {} \; -print
This iterates over the directories, and checks each one for the required file as well as making sure the forbidden file is not present.
An example, where only foo/bar3 contains yes.js without also containing no.js.
$ mkdir -p foo/bar1 foo/bar2 foo/bar3
$ touch foo/bar1/no.js foo/bar1/yes.js foo/bar2/no.js foo/bar3/yes.js
$ find foo -type d -exec sh -c 'test -e "$1"/yes.js && ! test -e "$1"/no.js' _ {} \; -print
foo/bar3
Passing this to your script requires some creative quoting, though:
somescript "find foo -type d -exec sh -c 'test -e \"\$1\"/yes.js && ! test -e \"\$1\"/no.js' _ {} \; -print"
If you are using bash, you can simplify it a little:
somescript $'find foo -type d -exec sh -c \'test -e "$1"/yes.js && ! test -e "$1"/no.js\' _ {} \; -print'
If you are willing to use the obsolete and possibly unsupported operator -a, you can reduce this to a single invocation of test.
find . -type d -exec test -e {}/mustExist.js -a ! -e {}/cannotExist.js \; -print
This is also a little simpler to pass to your script, since it does not itself contain any quotes:
somescript 'find . -type d -exec test -e {}/mustExist.js -a ! -e {}/cannotExist.js \; -print'
You can simplify it using multiple -exec primaries as well:
somescript 'find . -type d -exec test -e {}/mustExist.js \; ! -exec test -e {}/cannotExist.js \; -print'
which is a little less efficient (it runs test twice instead of once) but is more portable, while being easier to quote than the version that passes the string to sh -c.

bash: How to delimit strings to find files

What syntax should I use in a bash script to list files based on 3 dynamic values:
- older than X days
- in a specified directory
- whose name contains a specified string?
FILEAGE=7
FILEDIR='"/home/ecom/tmp"'
FILESTRING='"search-results-*"'
FILES_FOR_REMOVAL=$("/usr/bin/find "${FILEDIR}" -maxdepth 1 -type f -mtime +${FILEAGE} -name "${FILESTRING}" -exec ls -lth {} \;")
echo ${FILES_FOR_REMOVAL}
If I try the above I get:
-bash: /usr/bin/find "/home/ecom/tmp" -maxdepth 1 -type f -mtime +7 -name "search-results-*" -exec ls -lth {} \;: No such file or directory
Remove superfluous quotes:
FILEAGE=7
FILEDIR='/home/ecom/tmp'
FILESTRING='search-results-*'
FILES_FOR_REMOVAL=$(/usr/bin/find "${FILEDIR}" -maxdepth 1 -type f -mtime +${FILEAGE} -name "${FILESTRING}" -exec ls -lth {} \;)
Your syntax for 'find' looks ok. Try removing the quotes around the command string, i.e.
FILES_FOR_REMOVAL=$(/usr/bin/find "${FILEDIR}" -maxdepth 1 -type f -mtime +${FILEAGE} -name "${FILESTRING}" -exec ls -lth {} \;)
FILEAGE=7
FILEDIR='/home/ecom/tmp'
FILESTRING='search-results-*'
/usr/bin/find "${FILEDIR}" -maxdepth 1 -type f -mtime +${FILEAGE} -name "${FILESTRING}" -exec /bin/ls -lth '{}' \;
There were some extra quotes that created the error. Also specify full path to /bin/ls to avoid problems with potential aliasing of ls(1). And to get filenames on a separate line, I dropped the $FILES_FOR_REMOVAL variable. You can also use
/usr/bin/find "${FILEDIR}" -maxdepth 1 -type f -mtime +${FILEAGE} -name "${FILESTRING}" -ls
(I can't add comments, but ... )
To reliably handle file names with spaces, you may want to consider storing the file list in a temp text file instead of a variable and loop through it using a while construct (instead of a for)
For example:
FILEAGE=7
FILEDIR='/home/ecom/tmp'
FILESTRING='search-results-*'
TEMPFILE=".temp${RANDOM}"
CMD="find \"${FILEDIR}\" -maxdepth 1 -type f -mtime +${FILEAGE} -name \"${FILESTRING}\" -exec /bin/ls -lth '{}' \;"
$CMD > $TEMPFILE # write output to file
while read thefile; do
do_somthing_to $thefile
done < $TEMPFILE
rm $TEMPFILE # clean up after
Or, if you're only going to use the list once, pipe the output directly to the while construct:
$CMD | while read thefile; do
do_something_to $thefile
done

Resources