I have a directory (output) in unix (SUN). There are two types of files created with timestamp prefix to the file name. These file are created on a regular interval of 10 minutes.
e. g:
1. 20140129_170343_fail.csv (some lines are there)
2. 20140129_170343_success.csv (some lines are there)
Now I have to search for a particular string in all the files present in the output directory and if the string is found in fail and success files, I have to count the number of lines present in those files and save the output to the cnt_succ and cnt_fail variables. If the string is not found I will search again in the same directory after a sleep timer of 20 seconds.
here is my code
#!/usr/bin/ksh
for i in 1 2
do
grep -l 0140127_123933_part_hg_log_status.csv /osp/local/var/log/tool2/final_logs/* >log_t.txt; ### log_t.txt will contain all the matching file list
while read line ### reading the log_t.txt
do
echo "$line has following count"
CNT=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT=`expr $CNT - 1`
echo $CNT
done <log_t.txt
if [ $CNT > 0 ]
then
exit
fi
echo "waiitng"
sleep 20
done
The problem I'm facing is, I'm not able to get the _success and _fail in file in line and and check their count
I'm not sure about ksh, but while ... do; ... done is notorious for running off with whatever variables you're using in bash. ksh might be similar.
If I've understand your question right, SunOS has grep, uniq and sort AFAIK, so a possible alternative might be...
First of all:
$ cat fail.txt
W34523TERG
ADFLKJ
W34523TERG
WER
ASDTQ34T
DBVSER6
W34523TERG
ASDTQ34T
DBVSER6
$ cat success.txt
abcde
defgh
234523452
vxczvzxc
jkl
vxczvzxc
asdf
234523452
vxczvzxc
dlkjhgl
jkl
wer
234523452
vxczvzxc
And now:
egrep "W34523TERG|ASDTQ34T" fail.txt | sort | uniq -c
2 ASDTQ34T
3 W34523TERG
egrep "234523452|vxczvzxc|jkl" success.txt | sort | uniq -c
3 234523452
2 jkl
4 vxczvzxc
Depending on the input data, you may want to see what options sort has on your system. Examining uniq's options may prove useful too (it can do more than just count duplicates).
Think you want something like this (will work in both bash and ksh)
#!/bin/ksh
while read -r file; do
lines=$(wc -l < "$file")
((sum+=$lines))
done < <(grep -Rl --include="[1|2]*_fail.csv" "somestring")
echo "$sum"
Note this will match files starting with 1 or 2 and ending in _fail.csv, not exactly clear if that's what you want or not.
e.g. Let's say I have two files, one starting with 1 (containing 4 lines) and one starting with 2 (containing 3 lines), both ending in `_fail.csv somewhere under my current working directory
> abovescript
7
Important to understand grep options here
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all
symbolic links, unlike -r.
and
-l, --files-with-matches
Suppress normal output; instead print the name of each input
file from which output would normally have been printed. The
scanning will stop on the first match. (-l is specified by
POSIX.)
Finaly I'm able to find the solution. Here is the complete code:
#!/usr/bin/ksh
file_name="0140127_123933.csv"
for i in 1 2
do
grep -l $file_name /osp/local/var/log/tool2/final_logs/* >log_t.txt;
while read line
do
if [ $(echo "$line" |awk '/success/') ] ## will check the success file
then
CNT_SUCC=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT_SUCC=`expr $CNT_SUCC - 1`
fi
if [ $(echo "$line" |awk '/fail/') ] ## will check the fail file
then
CNT_FAIL=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT_FAIL=`expr $CNT_FAIL - 1`
fi
done <log_t.txt
if [ $CNT_SUCC > 0 ] && [ $CNT_FAIL > 0 ]
then
echo " Fail count = $CNT_FAIL"
echo " Success count = $CNT_SUCC"
exit
fi
echo "waitng for next search..."
sleep 10
done
Thanks everyone for your help.
I don't think I'm getting it right, but You can't diffrinciate the files?
maybe try:
#...
CNT=`expr $CNT - 1`
if [ $(echo $line | grep -o "fail") ]
then
#do something with fail count
else
#do something with success count
fi
Related
I am trying to loop through all the files in a folder and add the file name of those files with 10 lines to a txt file but I don't know how to write the if statement.
As of right now, what I have is:
for FILE in *.txt do if wc $FILE == 10; then "$FILE" >> saved_names.txt fi done
I am getting stuck in how to format the statement that will evaluate to a boolean for the if statement.
I have already tried the if statement as:
if [ wc $FILE != 10 ]
if "wc $FILE" != 10
if "wc $FILE != 10"
as well as other ways but I don't seem to get it right. I know I am new to Bash but I can't seem to find a solution to this question.
There are a few problems in your code.
To count the number of lines in the file you should run "wc -l" command. However, that command will result in the number of lines and the name of the file (so for example - 10 a.txt - you can test it by running the command on a file in your terminal). To receive only the number of lines you need to pass the file's name to the standard input of that command
"==" is used in bash to compare strings. To compare integers as in that case, you should use "-eq" (take a look here https://tldp.org/LDP/abs/html/comparison-ops.html)
In terms of brackets: To get the wc command result you need to run it in a terminal and switch the command in the code to the result. To do that, you need correct brackets - $(wc -l). To receive a result of the comparison as a bool, you need to use square brackets with spaces [ 1 -eq 1 ].
To save the name of the file in another file using >> you need to first put the name to the standard output (as >> redirect the standard output to the chosen place). To do that you can just use the echo command.
The code should look like this:
#!/bin/bash
for FILE in *.txt
do
if [ "$(wc -l < "$FILE")" -eq 10 ]
then
echo "$FILE" >> saved_names.txt
fi
done
Try:
for file in *.txt; do
if [[ $(wc -l < "$file") -eq 10 ]]; then
printf '%s\n' "$file"
fi
done > saved_names.txt
Change > to >> if you want to append the filenames.
Related docs:
Command Substitution
Conditional Constructs
Extract the actual number of lines from a file with wc -l $FILE | cut -f1 -d' ' and use -eq operator:
for FILE in *.txt; do if [ "$(wc -l $FILE | cut -f1 -d' ')" -eq 10 ]; then "$FILE" >> saved_names.txt; fi; done
I am creating a script to help me through my daily work and automate it. I have encountered my problem when trying to input multiple lines in my while loop. I usually do it in my for loop but I execute it via command.
Sample:
for i in `cat listoffiles.txt`
do
echo $i
find <path> -name *$i* | awk -F "." {'print $4'} #to display a specific value
done
Now I am trying to automate it with a while loop. Having problems to read multiple input lines in it.
For example:
i want to search for these inputs:
For
Example
only
here is my script for it:
#!/bin/ksh
echo Please enter file #:
read Var1
while true
do
VarSession=`find $OT_DIR/archive*/ -name *$Var1* | awk -F "." {'print $4'}`
if [ "$VarSession" = "" ]
then
echo No match for File# $Var1 on this leg or is out of retention.
else
echo File# $Var1 is under Session# $VarSession
fi
done
VarSession=`find $OT_DIR/archive*/ -name *$Var1* | awk -F "." {'print $4'}`
Assuming that you provide 1 2 3 as input, The line above translates to this
VarSession=`find $OT_DIR/archive*/ -name "1 2 3" | awk -F "." {'print $4'}`
But you want to search all those values separately so you need another loop. for loop serves the purpose if traversing white-space separated entries.
Also, based upon the original script that you showed, I assume you want the script to search file by file, rather than scanning entire directories. However, the statement above will put all output in the variable without traversing it. To traverse line by line, while loop does the job.
#!/bin/ksh
# -n switch suppresses printing a newline
echo -n 'Please enter file #: '
read Var1
# Traverse over all entered values in Var1 (separated by white space)
for i in $Var1
do
#Set a flag to zero, logic explained later
Flag=0
find $OT_DIR/archive*/ -name *$i* | while read FileName
do
#Set the Flag to 1 if find command finds something
Flag=1
VarSession=`echo $FileName | awk -F "." {'print $4'}`
if [ "$VarSession" = "" ]
then
#If find found a file but VarSession has nothing then file name is not correct
echo "Some conventions went wrong in file name: $FileName"
else
echo "File# $Var1 is under Session# $VarSession"
fi
done
#If find found nothing, there was no match
if [ $Flag -eq 0 ]
then
echo No match for File# $Var1 on this leg or is out of retention.
fi
done
I need to check if a file has more than 1 line. I tried this:
if [ `wc -l file.txt` -ge "2" ]
then
echo "This has more than 1 line."
fi
if [ `wc -l file.txt` >= 2 ]
then
echo "This has more than 1 line."
fi
These just report errors. How can I check if a file has more than 1 line in a BASH conditional?
The command:
wc -l file.txt
will generate output like:
42 file.txt
with wc helpfully telling you the file name as well. It does this in case you're checking out a lot of files at once and want individual as well as total stats:
pax> wc -l *.txt
973 list_of_people_i_must_kill_if_i_find_out_i_have_cancer.txt
2 major_acheivements_of_my_life.txt
975 total
You can stop wc from doing this by providing its data on standard input, so it doesn't know the file name:
if [[ $(wc -l <file.txt) -ge 2 ]]
The following transcript shows this in action:
pax> wc -l qq.c
26 qq.c
pax> wc -l <qq.c
26
As an aside, you'll notice I've also switched to using [[ ]] and $().
I prefer the former because it has less issues due to backward compatibility (mostly to do with with string splitting) and the latter because it's far easier to nest executables.
A pure bash (≥4) possibility using mapfile:
#!/bin/bash
mapfile -n 2 < file.txt
if ((${#MAPFILE[#]}>1)); then
echo "This file has more than 1 line."
fi
The mapfile builtin stores what it reads from stdin in an array (MAPFILE by default), one line per field. Using -n 2 makes it read at most two lines (for efficiency). After that, you only need to check whether the array MAPFILE has more that one field. This method is very efficient.
As a byproduct, the first line of the file is stored in ${MAPFILE[0]}, in case you need it. You'll find out that the trailing newline character is not trimmed. If you need to remove the trailing newline character, use the -t option:
mapfile -t -n 2 < file.txt
if [ `wc -l file.txt | awk '{print $1}'` -ge "2" ]
...
You should always check what each subcommand returns. Command wc -l file.txt returns output in the following format:
12 file.txt
You need first column - you can extract it with awk or cut or any other utility of your choice.
How about:
if read -r && read -r
then
echo "This has more than 1 line."
fi < file.txt
The -r flag is needed to ensure line continuation characters don't fold two lines into one, which would cause the following file to report one line only:
This is a file with _two_ lines, \
but will be seen as one.
change
if [ `wc -l file.txt` -ge "2" ]
to
if [ `cat file.tex | wc -l` -ge "2" ]
If you're dealing with large files, this awk command is much faster than using wc:
awk 'BEGIN{x=0}{if(NR>1){x=1;exit}}END{if(x>0){print FILENAME,"has more than one line"}else{print FILENAME,"has one or less lines"}}' file.txt
I want to find for each function defined in a C source file how many times it's called and on which line.
Should I search for patterns which look like function definitions in C and then count how many times that function name occurs. If so, how can I do it? regular expressions?
Any help will be highly appreciated!
#!/bin/bash
if [ -r $1 ]; then
#??????
else
echo The file \"$1\" does NOT exist
fi
The final result is: (please report any bugs)
10 if [ -r $1 ]; then
11 functs=`grep -n -e "\(void\|double\|char\|int\) \w*(.*)" $1 | sed 's/^.*\(void\|double\|int\) \(\w*\)(.*$/\2/g'`
12 for f in $functs;do
13 echo -n $f\(\) is called:
14 grep -n $f $1 > temp.txt
15 echo -n `grep -c -v -e "\(void\|double\|int\) $f(.*)" -e"//" temp.txt`
16 echo " times"
17 echo -n on lines:
18 echo -n `grep -v -e "\(void\|double\|int\) $f(.*)" -e"//" temp.txt | sed -n 's/^\([0-9]*\)[:].*/\1/p'`
19 echo
20 echo
21 done
22 else
23 echo The file \"$1\" does not exist
24 fi
This might sort of work. The first bit finds function definitions like
<datatype> <name>(<stuff>)
and pulls out the <name>. Then grep for that string. There are loads of situations where this won't work, but it might be a good place to start if you're trying to make a simple shell script that works on some programs.
functions=`grep -e "\(void\|double\|int\) \w*(.*)$" -f input.c | sed 's/^.*\(void\|double\|int\) \(\w*\)(.*$/\2/g'`
for func in $functions
do
echo "Counting references for $func:"
grep "$func" -f input.c | wc -l
done
You can try with this regex
(^|[^\w\d])?(functionName(\s)*\()
for example to search all printf occurrences
(^|[^\w\d])?(printf(\s)*\()
to use this expression with grep you have to use the option -E, like this
grep -E "(^|[^\w\d])?(printf(\s)*\()" the_file.txt
Final note, what miss with this solution is to skip the occurrences in comment bloks.
What is the right way to test if several variables are all equal?
if [[ $var1 = $var2 = $var3 ]] # syntax error
Is it necessary to write something like the following?
if [[ $var1 = $var2 && $var1 = $var3 && $var2 = $var3 ]] # cumbersome
if [[ $var1 = $var2 && $var2 = $var3 && $var3 = $var4 ]] # somewhat better
Unfortunately, the otherwise excellent Advanced Bash Scripting Guide and other online sources I could find don't provide such an example.
My particular motivation is to test if several directories all have the same number of files, using ls -1 $dir | wc -l to count files.
Note
"var1" etc. are example variables. I'm looking for a solution for arbitrary variable names, not just those with a predictable numeric ending.
Update
I've accepted Richo's answer, as it is the most general. However, I'm actually using Kyle's because it's the simplest and my inputs are guaranteed to avoid the caveat.
Thanks for the suggestions, everyone.
if you want to test equality of an arbitrary number of items (let's call them $item1-5, but they could be an array
st=0
for i in $item2 $item3 $item4 $item5; do
[ "$item1" = "$i" ]
st=$(( $? + st ))
done
if [ $st -eq 0 ]; then
echo "They were all the same"
fi
If they are single words you can get really cheap about it.
varUniqCount=`echo "${var1} ${var2} ${var3} ${var4}" | sort -u | wc -l`
if [ ${varUniqCount} -gt 1 ]; then
echo "Do not match"
fi
Transitive method of inspection.
#!/bin/bash
var1=10
var2=10
var3=10
if [[ ($var1 == $var2) && ($var2 == $var3) ]]; then
echo "yay"
else
echo "nay"
fi
Output:
[jaypal:~/Temp] ./s.sh
yay
Note:
Since you have stated in your question that your objective is to test several directories that have same number of files, I thought of the following solution. I know this isn't something you had request so please feel free to disregard it.
Step1:
Identify number of files in a given directory. This command will look inside sub-dirs too but that can be controlled using -depth option of find.
[jaypal:~/Temp] find . -type d -exec sh -c "printf {} && ls -1 {} | wc -l " \;
. 9
./Backup 7
./bash 2
./GTP 22
./GTP/ParserDump 11
./GTP/ParserDump/ParserDump 1
./perl 7
./perl/p1 2
./python 1
./ruby 0
./scripts 22
Step2:
This can be combined with Step1 as we are just redirecting the content to a file.
[jaypal:~/Temp] find . -type d -exec sh -c "printf {} && ls -1 {} | wc -l " \; > file.temp
Step3:
Using the following command we will look in the file.temp twice and it will give us a list of directories that have same number of files.
[jaypal:~/Temp] awk 'NR==FNR && a[$2]++ {b[$2];next} ($2 in b)' file.temp file.temp | sort -k2
./GTP/ParserDump/ParserDump 1
./python 1
./bash 2
./perl/p1 2
./Backup 7
./perl 7
./GTP 22
./scripts 22
(edited to include delimiters to fix the problem noted by Keith Thompson)
Treating the variable values as strings, you can concatenate them along with a suitable delimiter and do one comparison:
if [[ "$var1|$var2|$var3" = "$var1|$var1|$var1" ]]
I used = instead == because == isn't an equality comparison inside [[ ]], it is a pattern match.
For your specific case, this should work:
distinct_values=$(for dir in this_dir that_dir another_dir ; do ls -l "$dir" | wc -l ; done | uniq | wc -l)
if [ $distinct_values -eq 1 ] ; then
echo All the same
else
echo Not all the same
fi
Explanation:
ls -l "$dir" lists the files and subdirectories in the directory, one per line (omitting dot files).
Piping the output through wc -l gives you the number of files in the directory.
Doing that consecutively for each directory in the list gives you a list consisting of the number of files in each directory; if there are 7 in each, this gives 3 lines each consisting of the number 7
Piping that through uniq eliminates consecutive duplicate lines.
Piping that through wc -l gives you the number of distinct lines, which will be 1 if and only if all the directories contain the same number of files.
Note that the output of the 4th stage doesn't necessarily give you the number of distinct numbers of files in the directories; uniq only removes adjacent duplicates, so if the inputs are 7 6 7, the two 7s won't be merged. But it will merge all lines into 1 only if they're all the same.
This is the power of the Unix command line: putting small tools together to do interesting and useful things. (Show me a GUI that can do that!)
For values stored in variables, replace the first line by:
distinct_values=$(echo "$this_var" "$that_var" "$another_var" | fmt -1 | uniq | wc -l)
This assumes that the values of the variables don't contain spaces.