This question already has answers here:
Variables as commands in Bash scripts
(5 answers)
Closed 3 years ago.
I'm trying to count the number of files with different extensions in /foo/.
case 1 works as expected, but more flexible situations such as case 2 or case 3 don't work as expected.
File test.sh
# case 1
vista=$(find /foo/*.zip -atime -1)
echo "$vista" | wc -l
# case 2
vista=$(find /foo/*)
echo "$vista.zip -atime -1" | wc -l
# case 3
echo "$vista.xz -atime -1" | wc -l
Output:
./test.sh
187
4566
4566
I suspect the problem is that for example echo "$vista.zip -atime -1" from case 2 runs first find /foo/* before appending the string zip -atime -1, but I don't know how to do it right.
Code should never be stored in strings (unless using printf %q to generate eval-safe versions of variables, and then using eval at runtime). Use either an array (for dynamically-constructed content) or a function.
The former:
find_cmd=( find /foo/* )
"${find_cmd[#]}" -atime -1 | wc -l
The latter:
find_under_foo() { find /foo/* "$#"; }
find_under_foo -atime -1 | wc -l
Related
This question already has answers here:
How to pass the value of a variable to the standard input of a command?
(9 answers)
Closed 1 year ago.
im running this in bashrc file
function simplefunc () {
output=$(ls -1 "$HOME")
linecount=$(${output} | wc -l)
echo "${linecount}"
echo "${output}"
}
getting this error
Desktop: command not found
0
Desktop
Documents
Downloads
Music
Pictures
Public
snap
SoftMaker
Templates
venv
Videos
i tried these too
putting "local" before variable or
# linecount=$(output | wc -l)
# echo "$(${output} | wc -l)"
I think you should change the third line to:
linecount="$(echo "${output}" | wc -l)"
# OR alternatvely:
# linecount="$(wc -l < <(echo "$output"))"
This question already has answers here:
When to wrap quotes around a shell variable?
(5 answers)
Compare files with each other within the same directory
(6 answers)
Closed 4 years ago.
Answer to my question using Kubator command line :
#Function that shows the files having the same content in the current directory
showDuplicates (){
last_file=''
while read -r f1_hash f1_name; do
if [ "$last_file" != "$f1_hash" ]; then
echo "The following files have the exact same content :"
echo "$f1_name"
while read -r f2_hash f2_name; do
if [ "$f1_hash" == "$f2_hash" ] && [ "$f1_name" != "$f2_name" ]; then
echo "$f2_name"
fi
done < <(find ./ -maxdepth 1 -type f -print0 | xargs -0 md5sum | sort -k1,32 | uniq -w32 -D)
fi
last_file="$f1_hash"
done < <(find ./ -maxdepth 1 -type f -print0 | xargs -0 md5sum | sort -k1,32 | uniq -w32 -D)
}
Original question :
I've seen some discussions about what I'm about to ask but I have troubles understanding the mechanics behind the solution proposed and I have not been able to solve my problem that follows.
I want to make a function to compare files, for that, naively, I've tried the following :
#somewhere I use that to get the files paths
files_to_compare=$(find $base_path -maxdepth 1 -type f)
files_to_compare=( $files_to_compare )
#then I pass files_to_compare as an argument to the following function
showDuplicates (){
files_to_compare=${1}
n_files=$(( ${#files_to_compare[#]} ))
for (( i=0; i < $n_files ; i=i+1 )); do
for (( j=i+1; j < $n_files ; j=j+1 )); do
sameContent "${files_to_compare[i]}" "${files_to_compare[j]}"
r=$?
if [ $r -eq 1 ]; then
echo "The following files have the same content :"
echo ${files_to_compare[i]}
echo ${files_to_compare[j]}
fi
done
done
}
The function 'sameContent' takes the absolute paths of two files and makes use of different commends (du, wc, diff) to return 1 or 0 depending on the files having the same content (respectively).
The incorrectness of that code has showed up with file names containing spaces but I've since read that it's not the way to go to manipulate files in bash.
On https://unix.stackexchange.com/questions/392393/bash-moving-files-with-spaces and some other pages I've read that the correct way to go is to use a code that looks like this :
$ while IFS= read -r file; do echo "$file"; done < files
I seem not to be able to understand what lies behind that bit of code and how I could use it to solve my problem. Particularly due to the fact that I want/need to use intricate loops.
I'm new to bash and it's seems to be a common problem but still if someone was kind enough to give me some insight about how that works that would be wonderful.
p.s.: please excuse the probable grammar mistakes
How about to use md5sum to compare content of files in Your folder instead. That's way safer and standard way. Then You would need only something like this:
find ./ -type f -print0 | xargs -0 md5sum | sort -k1,32 | uniq -w32 -D
What it does:
find finds all files -type f in current folder ./ and output separates by null byte -print0 that's needed for special characters like space in filenames (like You are mentioning moving files with space)
xargs takes output from find separated by null byte -0 and performs md5sum hashes on files
sort sorts output by positions 1-32 (which is md5 hash) -k1,32
uniq makes output unique by first 32 characters (md5 hash) -w32 and filter only duplicated lines -D
Output example:
7a2e203cec88aeffc6be497af9f4891f ./file1.txt
7a2e203cec88aeffc6be497af9f4891f ./folder1/copy_of_file1.txt
e97130900329ccfb32516c0e176a32d5 ./test.log
e97130900329ccfb32516c0e176a32d5 ./test_copy.log
If performance is crucial this can be tuned to sort firstly by filesize and only then compare md5sums. Or called mv, rm etc.
I'm writing a bash script that checks the number of files in a directory and if it's over a certain number, do something. The code is here:
DIR=/home/inventory
waiting="$((ls ${DIR}/waiting/ -1 | wc -l))"
echo $waiting
if [ $waiting -gt 3 ]
then
(DO SOME STUFF HERE)
fi
The error I am getting is this line....
waiting="$((ls ${DIR}/waiting/ -1 | wc -l))"
Specifically the error is ....
division by 0 (error token is "/home/inventory/waiting/ -1 | wc -l")
I thought trying to put the number of files in this directory into a variable would work using $(()).
Does anyone have an idea why this is failing?
Many TIA.....
Jane
Use single parenthesis:
waiting="$(ls ${DIR}/waiting/ -1 | wc -l)"
$(( ... )) is used to perform arithmetic calculations.
From the man page:
((expression))
The expression is evaluated according to the rules described below under ARITHMETIC EVALUATION. If the value of the expression is non-zero, the return status is 0; otherwise the return status is 1. This is exactly equivalent to let "expression".
It is not recommended to use output of ls to count number of files/directories in a sub -directory as file names may contain newlines or glob characters as well.
Here is one example of doing it safely using gnu find:
dir=/home/inventory
waiting=$(find "$dir" -mindepth 1 -maxdepth 1 -printf '.' | wc -c)
If you don't have gnu find then use:
waiting=$(find "$dir" -mindepth 1 -maxdepth 1 -exec printf '.' \; | wc -c)
Another alternative is to pipe the waiting variable through awk to check for a value greater than 3 so:
DIR=/home/inventory
waiting="$((ls ${DIR}/waiting/ -1 | wc -l))"
echo $waiting
res=$(echo $waiting | awk '{ if ( $0 > 3 ) print "OK" }')
if [ $res == "OK" ]
then
(DO SOME STUFF HERE)
fi
In case someone has a similar problem even without using the compound command ((..))
Be sure your variables are not defined using let keyword, as it would force shell arithmetic evaluation too.
From the referece:
The shell allows arithmetic expressions to be evaluated, as one of the
shell expansions or by using the (( compound command, the let builtin,
or the -i option to the declare builtin.
I have a bash script where I'm trying to find out the number of files in a directory and perform an addition operation on it as well.
But while doing the same I'm getting the error as follows:
admin> ./fileCount.sh
1
./fileCount.sh: line 6: 22 + : syntax error: operand expected (error token is " ")
My script is as shown:
#!/usr/bin/bash
Var1=22
Var2= ls /stud_data/Input_Data/test3 | grep ".txt" | wc -l
Var3= $(($Var1 + $Var2))
echo $Var3
Can anyone point out where is the error.
A little away
As #devnull already answered to the question point out where is the error,
Just some more ideas:
General unix
To make this kind of browsing, there is a very powerfull command find that let you find recursively, exactly what you're serching for:
Var2=`find /stud_data/Input_Data/test3 -name '*.txt' | wc -l`
If you won't this to be recursive:
Var2=`find /stud_data/Input_Data/test3 -maxdepth 1 -name '*.txt' | wc -l`
If you want files only (meaning no symlink, nor directories)
Var2=`find /stud_data/Input_Data/test3 -maxdepth 1 -type f -name '*.txt' | wc -l`
And so on... Please read the man page: man find.
Particular bash solutions
As your question stand for bash, there is some bashism you could use to make this a lot quicker:
#!/bin/bash
Var1=22
VarLs=(/stud_data/Input_Data/test3/*.txt)
[ -e $VarLs ] && Var2=${#VarLs[#]} || Var2=0
Var3=$(( Var1 + Var2 ))
echo $Var3
# Uncomment next line to see more about current environment
# set | grep ^Var
Where bash expansion will translate /path/*.txt in an array containing all filenames matching the jocker form.
If there is no file matching the form, VarLs will only contain the jocker form himself.
So the test -e will correct this: If the first file of the returned list exist, then assing the number of elements in the list (${#VarLs[#]}) to Var2 else, assign 0 to Var2.
Can anyone point out where is the error.
You shouldn't have spaces around =.
You probably wanted to use command substitution to capture the result in Var2.
Try:
Var1=22
Var2=$(ls /stud_data/Input_Data/test3 | grep ".txt" | wc -l)
Var3=$(($Var1 + $Var2))
echo $Var3
Moreover, you could also say
Var3=$((Var1 + Var2))
What is the right way to test if several variables are all equal?
if [[ $var1 = $var2 = $var3 ]] # syntax error
Is it necessary to write something like the following?
if [[ $var1 = $var2 && $var1 = $var3 && $var2 = $var3 ]] # cumbersome
if [[ $var1 = $var2 && $var2 = $var3 && $var3 = $var4 ]] # somewhat better
Unfortunately, the otherwise excellent Advanced Bash Scripting Guide and other online sources I could find don't provide such an example.
My particular motivation is to test if several directories all have the same number of files, using ls -1 $dir | wc -l to count files.
Note
"var1" etc. are example variables. I'm looking for a solution for arbitrary variable names, not just those with a predictable numeric ending.
Update
I've accepted Richo's answer, as it is the most general. However, I'm actually using Kyle's because it's the simplest and my inputs are guaranteed to avoid the caveat.
Thanks for the suggestions, everyone.
if you want to test equality of an arbitrary number of items (let's call them $item1-5, but they could be an array
st=0
for i in $item2 $item3 $item4 $item5; do
[ "$item1" = "$i" ]
st=$(( $? + st ))
done
if [ $st -eq 0 ]; then
echo "They were all the same"
fi
If they are single words you can get really cheap about it.
varUniqCount=`echo "${var1} ${var2} ${var3} ${var4}" | sort -u | wc -l`
if [ ${varUniqCount} -gt 1 ]; then
echo "Do not match"
fi
Transitive method of inspection.
#!/bin/bash
var1=10
var2=10
var3=10
if [[ ($var1 == $var2) && ($var2 == $var3) ]]; then
echo "yay"
else
echo "nay"
fi
Output:
[jaypal:~/Temp] ./s.sh
yay
Note:
Since you have stated in your question that your objective is to test several directories that have same number of files, I thought of the following solution. I know this isn't something you had request so please feel free to disregard it.
Step1:
Identify number of files in a given directory. This command will look inside sub-dirs too but that can be controlled using -depth option of find.
[jaypal:~/Temp] find . -type d -exec sh -c "printf {} && ls -1 {} | wc -l " \;
. 9
./Backup 7
./bash 2
./GTP 22
./GTP/ParserDump 11
./GTP/ParserDump/ParserDump 1
./perl 7
./perl/p1 2
./python 1
./ruby 0
./scripts 22
Step2:
This can be combined with Step1 as we are just redirecting the content to a file.
[jaypal:~/Temp] find . -type d -exec sh -c "printf {} && ls -1 {} | wc -l " \; > file.temp
Step3:
Using the following command we will look in the file.temp twice and it will give us a list of directories that have same number of files.
[jaypal:~/Temp] awk 'NR==FNR && a[$2]++ {b[$2];next} ($2 in b)' file.temp file.temp | sort -k2
./GTP/ParserDump/ParserDump 1
./python 1
./bash 2
./perl/p1 2
./Backup 7
./perl 7
./GTP 22
./scripts 22
(edited to include delimiters to fix the problem noted by Keith Thompson)
Treating the variable values as strings, you can concatenate them along with a suitable delimiter and do one comparison:
if [[ "$var1|$var2|$var3" = "$var1|$var1|$var1" ]]
I used = instead == because == isn't an equality comparison inside [[ ]], it is a pattern match.
For your specific case, this should work:
distinct_values=$(for dir in this_dir that_dir another_dir ; do ls -l "$dir" | wc -l ; done | uniq | wc -l)
if [ $distinct_values -eq 1 ] ; then
echo All the same
else
echo Not all the same
fi
Explanation:
ls -l "$dir" lists the files and subdirectories in the directory, one per line (omitting dot files).
Piping the output through wc -l gives you the number of files in the directory.
Doing that consecutively for each directory in the list gives you a list consisting of the number of files in each directory; if there are 7 in each, this gives 3 lines each consisting of the number 7
Piping that through uniq eliminates consecutive duplicate lines.
Piping that through wc -l gives you the number of distinct lines, which will be 1 if and only if all the directories contain the same number of files.
Note that the output of the 4th stage doesn't necessarily give you the number of distinct numbers of files in the directories; uniq only removes adjacent duplicates, so if the inputs are 7 6 7, the two 7s won't be merged. But it will merge all lines into 1 only if they're all the same.
This is the power of the Unix command line: putting small tools together to do interesting and useful things. (Show me a GUI that can do that!)
For values stored in variables, replace the first line by:
distinct_values=$(echo "$this_var" "$that_var" "$another_var" | fmt -1 | uniq | wc -l)
This assumes that the values of the variables don't contain spaces.