This is my first Bash script so forgive me if this question is trivial. I need to count the number of files within a specified directory $HOME/.junk. I thought this would be simple and assumed the following would work:
numfiles= find $HOME/.junk -type f | wc -l
echo "There are $numfiles files in the .junk directory."
Typing find $HOME/.junk -type f | wc -l at the command line works exactly how I expected it to, simply returning the number of files. Why is this not working when it is entered within my script? Am I missing some special notation when it comes to passing options to the utilities?
Thank you very much for your time and help.
You just need to surround it with backticks:
numfiles=`find $HOME/.junk -type f | wc -l`
The term for this is command substitution.
if you are using bash you can also use $() for command substitution, like so:
numfiles=$(find $HOME/.junk -type f | wc -l)
I find this to be slightly more readable than backquotes, as well as having the ability to nest several commands inside one another.
with bash 4 (if you want recursive)
#!/bin/bash
shopt -s globstar
for file in **
do
((i++))
done
echo "total files: $i"
if not
#!/bin/bash
shopt -s dotglob
shopt -s nullglob
for file in *
do
((i++))
done
echo "total files: $i"
Related
This question already has answers here:
Loop over directories with whitespace in Bash
(3 answers)
Closed last year.
I am encountering strange behavior when executing the following bash script:
#! /bash/bin
dirs=$(ls .)
for dir in $dirs ; do
files=$(ls $dir)
for file in $files ; do
line=$(head -n -1 $dir/$file)
echo $line
done
done
Instead of the echo of $line, I am getting the results of a ls / command, followed by the echo of $line, followed by the contents of $files. My guess is that I am exceeding some limit in bash: I have 171 directories, each with 500 files. When I add the line:
echo ${#line}
to the script, I get the right answer (68); but the echo of $line fills my terminal window. What can I do now?
So, you want to print all but the last line of all files one directory level down the current level? As comments suggest, never parse the ls output, it is for humans, not for automation. Use wildcards, instead:
for f in */*; do
head -n -1 "$f"
done
However there are two potential issues:
There are no such files. By default there will be one iteration of the loop with variable f set to literal value */*. You can avoid this by enabling the nullglob option before the loop and disabling it again after the loop:
shopt -s nullglob
for f in */*; do
head -n -1 "$f"
done
shopt -u nullglob
Some "files" are not true files (links, directories...). But we can test this before executing head:
shopt -s nullglob
for f in */*; do
if [[ -f "$f" ]]; then
head -n -1 "$f"
fi
done
shopt -u nullglob
As noted by #JohnKugelman there is simpler than a loop:
head -n -1 */*
But unfortunately there is no simple way to solve the second issue mentioned above. So, if you are not 100% sure that all files in */* are really files, the loop with a test is safer...
... unless you use a dedicated utility like find:
find . -mindepth 2 -maxdepth 2 -type f -exec head -n -1 {} \;
The -type f test retains only real files. The -exec action takes one argument which is a command to execute on each found file, where {} stands for the path of the current file. This command must be terminated by an escaped semicolon (\;).
Note: if you have thousands of files and you print their full content but the last line, it is not surprising that it "fills your terminal window". Just in case what you want is to print only the first line of these files, replace head -n -1 by head -n1.
The answer is probably not related to the number of files and directories but to files that breaks the script behavior.
For example the script itself is a file which is not directory but will be listed in the first loop.
Any directory within the directories will also break the script.
Also notice the error in the first line which should be
#!/bin/bash
What I am trying to do is to count all the files in a directory using shell script.
For example, when execute the program,
./test.sh project
it should count all the files in the folder called "project".
But I am having trouble with the directory part.
What I have done so far is,
#!/bin/bash
directory=$1
count=ls $directory | wc -l
echo "$folder has $count files"
but it does not work... Can anyone blow up my confusion please?
Thanks!
You have an incorrect syntax while setting the count, for running nested commands in bash you need to use command-substitution using $(..) which runs the commands in a sub-shell and returns the restult
count=$(ls -- "$directory" | wc -l)
But never parse ls output in scripts for any purpose, use the more general purpose find command
find "$1" -maxdepth 1 -type f | wc -l
Check more about what $(..) form Wiki Bash Hackers - Command substitution
#!/bin/bash
directory=$1
count=`ls $directory | wc -l`
echo "$folder has $count files"
#!/bin/bash
outbound=/home/user/outbound/
putfile=DATA_FILE_PUT_*.CSV
cd $outbound
filecnt=0
for file in $putfile; do let filecnt=filecnt+1; done
echo "Filecount: " $filecnt
So this code works well when there are files located in the outbound directory. I can place files into the outbound path and as long as they match the putfile mask then the files are incremented as expected.
Where the problem comes in is if I run this while there are no files located in $outbound.
If there are zero files there $filecnt still returns a 1 but I'm looking to have it return a 0 if there are no files there.
Am I missing something simple?
Put set -x just below the #! line to watch what your script is doing.
If there is no matching file, then the wildcard is left unexpanded, and the loop runs once, with file having the value DATA_FILE_PUT_*.CSV.
To change that, set the nullglob option. Note that this only works in bash, not in sh.
shopt -s nullglob
putfile=DATA_FILE_PUT_*.CSV
for file in $putfile; do let filecnt=filecnt+1; done
Note that the putfile variable contains the wildcard pattern, not the list of file names. It might make more sense to put the list of matches in a variable instead. This needs to be an array variable, and you need to change the current directory first. The number of matching files is then the length of the array.
#!/bin/bash
shopt -s nullglob
outbound=/home/user/outbound/
cd "$outbound"
putfiles=(DATA_FILE_PUT_*.CSV)
echo "Filecount: " ${#putfiles}
If you need to iterate over the files, take care to protect the expansion of the array with double quotes, otherwise if a file name contains whitespace then it will be split over several words (and if a filename contains wildcard characters, they will be expanded).
#!/bin/bash
shopt -s nullglob
outbound=/home/user/outbound/
cd "$outbound"
putfiles=(DATA_FILE_PUT_*.CSV)
for file in "${putfiles[#]}"; do
echo "Processing $file"
done
You could test if file exists first
for file in $putfile; do
if [ -f "$file" ] ; then
let filecnt=filecnt+1
fi
done
Or look for your files with find
for file in $(find . -type f -name="$putfile"); do
let filecnt=filecnt+1
done
or simply (fixed)
filecnt=$(find . -type f -name "$putfile" | wc -l); echo $filecnt
This is because when no matches are found, bash by default expands the wildcard DATA_FILE_PUT_*.CSV to the word DATA_FILE_PUT_*.CSV and therefore you end up with a count of 1.
To disable this behavior, use shopt -s nullglob
Not sure why you need a piece of code here. Following one liner should do your job.
ls ${outbound}/${putfile} | wc -l
Or
find ${outbound} -maxdepth 1 -type f -name "${putfile}" | wc -l
Using bash, how can one get the number of files in a folder, excluding directories from a shell script without the interpreter complaining?
With the help of a friend, I've tried
$files=$(find ../ -maxdepth 1 -type f | sort -n)
$num=$("ls -l" | "grep ^-" | "wc -l")
which returns from the command line:
../1-prefix_blended_fused.jpg: No such file or directory
ls -l : command not found
grep ^-: command not found
wc -l: command not found
respectively. These commands work on the command line, but NOT with a bash script.
Given a file filled with image files formatted like 1-pano.jpg, I want to grab all the images in the directory to get the largest numbered file to tack onto the next image being processed.
Why the discrepancy?
The quotes are causing the error messages.
To get a count of files in the directory:
shopt -s nullglob
numfiles=(*)
numfiles=${#numfiles[#]}
which creates an array and then replaces it with the count of its elements. This will include files and directories, but not dotfiles or . or .. or other dotted directories.
Use nullglob so an empty directory gives a count of 0 instead of 1.
You can instead use find -type f or you can count the directories and subtract:
# continuing from above
numdirs=(*/)
numdirs=${#numdirs[#]}
(( numfiles -= numdirs ))
Also see "How can I find the latest (newest, earliest, oldest) file in a directory?"
You can have as many spaces as you want inside an execution block. They often aid in readability. The only downside is that they make the file a little larger and may slow initial parsing (only) slightly. There are a few places that must have spaces (e.g. around [, [[, ], ]] and = in comparisons) and a few that must not (e.g. around = in an assignment.
ls -l | grep -v ^d | wc -l
One line.
How about:
count=$(find .. -maxdepth 1 -type f|wc -l)
echo $count
let count=count+1 # Increase by one, for the next file number
echo $count
Note that this solution is not efficient: it spawns sub shells for the find and wc commands, but it should work.
file_num=$(ls -1 --file-type | grep -v '/$' | wc -l)
this is a bit lightweight than a find command, and count all files of the current directory.
The most straightforward, reliable way I can think of is using the find command to create a reliably countable output.
Counting characters output of find with wc:
find . -maxdepth 1 -type f -printf '.' | wc --char
or string length of the find output:
a=$(find . -maxdepth 1 -type f -printf '.')
echo ${#a}
or using find output to populate an arithmetic expression:
echo $(($(find . -maxdepth 1 -type f -printf '+1')))
Simple efficient method:
#!/bin/bash
RES=$(find ${SOURCE} -type f | wc -l)
Get rid of the quotes. The shell is treating them like one file, so it's looking for "ls -l".
REmove the qoutes and you will be fine
Expanding on the accepted answer (by Dennis W): when I tried this approach I got incorrect counts for dirs without subdirs in Bash 4.4.5.
The issue is that by default nullglob is not set in Bash and numdirs=(*/) sets an 1 element array with the glob pattern */. Likewise I suspect numfiles=(*) would have 1 element for an empty folder.
Setting shopt -s nullglob to disable nullglobbing resolves the issue for me. For an excellent discussion on why nullglob is not set by default on Bash see the answer here: Why is nullglob not default?
Note: I would have commented on the answer directly but lack the reputation points.
Here's one way you could do it as a function. Note: you can pass this example, dirs for (directory count), files for files count or "all" for count of everything in a directory. Does not traverse tree as we aren't looking to do that.
function get_counts_dir() {
# -- handle inputs (e.g. get_counts_dir "files" /path/to/folder)
[[ -z "${1,,}" ]] && type="files" || type="${1,,}"
[[ -z "${2,,}" ]] && dir="$(pwd)" || dir="${2,,}"
shopt -s nullglob
PWD=$(pwd)
cd ${dir}
numfiles=(*)
numfiles=${#numfiles[#]}
numdirs=(*/)
numdirs=${#numdirs[#]}
# -- handle input types files/dirs/or both
result=0
case "${type,,}" in
"files")
result=$((( numfiles -= numdirs )))
;;
"dirs")
result=${numdirs}
;;
*) # -- returns all files/dirs
result=${numfiles}
;;
esac
cd ${PWD}
shopt -u nullglob
# -- return result --
[[ -z ${result} ]] && echo 0 || echo ${result}
}
Examples of using the function :
folder="/home"
get_counts_dir "files" "${folder}"
get_counts_dir "dirs" "${folder}"
get_counts_dir "both" "${folder}"
Will print something like :
2
4
6
Short and sweet method which also ignores symlinked directories.
count=$(ls -l | grep ^- | wc -l)
or if you have a target:
count=$(ls -l /path/to/target | grep ^- | wc -l)
I started with this:
command *
But it doesn't work when the directory is empty; the * wildcard becomes a literal "*" character. So I switched to this:
for i in *; do
...
done
which works, but again, not if the directory is empty. I resorted to using ls:
for i in `ls -A`
but of course, then file names with spaces in them get split. I tried tacking on the -Q switch:
for i in `ls -AQ`
which causes the names to still be split, only with a quote character at the beginning and ending of the name. Am I missing something obvious here, or is this harder than it ought it be?
Assuming you only want to do something to files, the simple solution is to test if $i is a file:
for i in *
do
if test -f "$i"
then
echo "Doing somthing to $i"
fi
done
You should really always make such tests, because you almost certainly don't want to treat files and directories in the same way. Note the quotes around the "$i" which prevent problems with filenames containing spaces.
find could be what you want.
find . | while read file; do
# do something with $file
done
Or maybe like this:
find . -exec <command> {} \;
If you do not want the search to include subdirectories you might need to add a combination of -type f and -maxdepth 1 to the find command. See the find man page for details.
It depends whether you're going to type this at a command prompt, and which command you're applying to the files.
If it's typed you could go with your second choice and substitute something harmless for the command. I like to use echo instead of mv or rm, for example.
Put it all on one line:
for i in * ; do command $i; done
When that works - or you can see where it fails, and whether it's harmless, you can press up-arrow, edit the command and try again.
Use shopt to prevent expansion to *.txt
shopt -s nullglob
for myfile in *.txt
do
# your code here
echo $myfile
done
this should do the trick:
find -type d -print0 | xargs -n 1 -0 echo "your folder: {} !"
find -type f -print0 | xargs -n 1 -0 echo "your file: {} !"
the print0 / 0 are there to avoid problems with whitespace