Error while reading from a file in Unix - bash

I am new to Unix and need help with a certain code.
My if condition acts weird.
Suppose my data is as follows -
name1 place1 date1
name2 place2 date2
name3 place3 date3
name4 place4 date4
name5 place5 date5
This is what I had originally done -
SRC_PATH="/app/Informatica9.5.1/server/infa_shared/0149icadd/SrcFiles"
rm -f temp_con_total
cat $SRC_PATH/Control_Total.txt > temp_con_total
echo $SRC_PATH
i=0
while read line
do
echo $line
if [[[ $i -ge 0 ]]];
then
file_name=`echo -e $line | cut -d ' ' -f 2`
file_size=`echo -e $line | cut -d ' ' -f 3`
file_size=$(( $file_size ))
actual_size=`cat $SRC_PATH/$file_name | wc -l`
actual_size=`echo -e $actual_size | cut -d ' ' -f 1`
actual_size=$(( $actual_size - 1 ))
echo $file_name
echo $file_size
echo $actual_size
fi
i=$(( i + 1 ))
done < temp_con_total
in this case, it gave me only first 2 records.
Later, I removed one set of brackets from if condition -
if [ $i -ge 0 ];
in this case, it gave me only the first row.
later i put three brackets, as follows -
if [[[ $i -ge 0 ]]];
now, it did give me all rows but after each row it also gave me error as follows:
name1 place1 date1
loop.sh[10]: [[[: not found [No such file or directory]
name2 place2 date2
loop.sh[10]: [[[: not found [No such file or directory]
name3 place3 date3
loop.sh[10]: [[[: not found [No such file or directory]
name4 place4 date4
loop.sh[10]: [[[: not found [No such file or directory]
name5 place5 date5
loop.sh[10]: [[[: not found [No such file or directory]
Can someone please help me with this? Why is this happening?
loop.sh is the name of my script
What I need as output is, all rows in the file as follows-
name1 place1 date1
name2 place2 date2
name3 place3 date3
name4 place4 date4
name5 place5 date5
I want to know why is there an error after it prints each row? loop.sh[10]: [[[: not found [No such file or directory]
And how does the number of brackets affect the code?

Bash does not use square brackets. Not as brackets anyway. It looks like it does, but it doesn't.
There is a command called [ (that's its name, an opening bracket) that is much the same as the test command. That command takes a series of command line parameters, just like any other command, and it expects that the last parameter will be a closing bracket, ], but no actual bracket processing is done. This is why you always follow the closing bracket with a semi-colon, ;, to finish the command. If you look in /bin (or /usr/bin?) you'll see that /bin/[ exists as a file.
Likewise, there's another command called [[ (again, that's its name, two opening brackets together). This is the same idea as the [ command, but has slightly different features. This one doesn't usually have a file in /bin.
Both commands are implemented as bash builtins, for efficiency in the case of [ and because [[ has to be, I think, but that doesn't affect the syntax.
There is no command named [[[, hence "No such file or directory".
As for your script, the if is completely pointless because you test if $i is greater than, or equal to zero when there's no way it can't be. I think this should do what you want:
SRC_PATH="/app/Informatica9.5.1/server/infa_shared/0149icadd/SrcFiles"
rm -f temp_con_total
cp $SRC_PATH/Control_Total.txt temp_con_total
echo $SRC_PATH
while read line
do
echo $line
file_name=`echo -e $line | cut -d ' ' -f 2`
file_size=`echo -e $line | cut -d ' ' -f 3`
actual_size=`cat $SRC_PATH/$file_name | wc -l | cut -d ' ' -f 1`
(( actual_size-- ))
echo $file_name
echo $file_size
echo $actual_size
done < temp_con_total

Related

Bash Shellscript Column Check Error Handling

I am writing a Bash Shellscript. I need to check a file for if $value1 contains $value2. $value1 is the column number (1, 4, 5 as an example) and $value2 ($value2 can be '03', '04' , '09' etc) is the String I am looking for. If the column contains the $value2 then perform a move of the file to an error directory. I was wondering what is the best approach to this. I was thinking awk or is there another way?
$value1 and $value2 are stored in a config file. I have control over what format I can use. Here's an example. The file separator is Octal \036. I just depicted with | below.
Example
$value1=5
$value2=04
Input example1.txt
example|42|udajha|llama|04
example|22|udajha|llama|02
Input example2.txt
example|22|udajha|llama|02
Result
move example1.txt to /home/user/error_directory and example2.txt stays in current directory (nothing happens)
awk can report out which files meet this condition:
awk -F"|" -v columnToSearch=$value1 -v valueToFind=$value2 '$columnToSearch==valueToFind{print FILENAME}' example1.txt example2.txt
Then you can do your mv based on that.
Example using a pipe to xargs (with smaller variable names since you get the idea by now):
awk -F"|" -v c=$value1 -v v=$value2 '$c==v{print FILENAME}' example1.txt example2.txt | xargs -I{} mv -i {} /home/user/error_directory
If you're writing a bash shell script then you can break it down by column using cut.
There are really so many options that it depends on what you want to get done.
In my experience with data I'd use a colon rather than pipe because it allows me to avoid the escape with the 'cut' command.
Changing the data files to:
cat example1.txt
example:42:udajha:llama:04
example:22:udajha:llama:02
I'd write it like this: (adding -x so that you can see the processing, but in your code you'd not need to do that.)
[root#]# cat mysript.sh
#!/bin/sh -x
one=`cat example1.txt | cut -d: -f5`
two=`cat example2.txt | cut -d: -f5`
for i in $one
do
if [ $i -eq $two ]
then
movethis=`grep $two example1.txt`
echo $movethis >> /home/me/error.txt
fi
done
cat /home/me/error.txt
[root#]# ./mysript.sh
++ cat example1.txt
++ cut -d: -f5
+ one='04
02 '
++ cat example2.txt
++ cut -d: -f5
+ two=02
+ for i in '$one'
+ '[' 04 -eq 02 ']'
+ for i in '$one'
+ '[' 02 -eq 02 ']'
++ grep 02 example1.txt
+ movethis='example:22:udajha:llama:02 '
+ echo example:22:udajha:llama:02
+ cat /home/me/error.txt
example:22:udajha:llama:02
You can use any command you live to move your content. Touch, cp, mv, what ever you want to use there.

Assing a variable and use it inside of if statement shell scripting

I am new to shell scripting and trying to make a few small scripts. I got stuck when i tried to write if condition. In the code below i am trying to get the $5 value from df and trying to use it in if condition. However the code doesn't work.
#!/bin/sh
temp = $(df -h | awk '$NF=="/"{$5}')
if [ $temp > 60 ] ; then
df -h | awk '$NF=="/" {printf("%s\n"),$5}'
(date -d "today" +"Date:%Y.%m.%d"" Hour: %H:%M")
fi
#end
So i've figured out something and changed my code into this:
temp=$(df -h | awk '$NF=="/"{$5}')
if [ "$((temp))" -gt 0 ] ; then
df -h | awk '$NF=="/" {printf("%s\n"),$5}'
(date -d "today" +"Date:%Y.%m.%d"" Hour: %H:%M")
fi
#end
And now , i'm trying to get the integer value of the $5 variable. It returns a percentage and i want to compare this percentage with %60. How can i do that ?
Let's see what shellcheck.net has to tell us:
Line 1:
#!/bin/sh
^-- SC1114: Remove leading spaces before the shebang.
Line 3:
temp = $(df -h | awk '$NF=="/"{$5}')
^-- SC1068: Don't put spaces around the = in assignments.
Line 4:
if [ $temp > 0 ] ; then
^-- SC2086: Double quote to prevent globbing and word splitting.
^-- SC2071: > is for string comparisons. Use -gt instead.
^-- SC2039: In POSIX sh, lexicographical > is undefined.
Um, ok, after a little fixing:
#!/bin/sh
temp=$(df -h | awk '$NF=="/"{$5}')
if [ "$temp" -gt 0 ] ; then
df -h | awk '$NF=="/" {printf("%s\n"),$5}'
(date -d "today" +"Date:%Y.%m.%d"" Hour: %H:%M")
fi
The [ ... ] command is the same as test command. Test does not have < comparison for numbers. It has -gt (greater then). See man test.
This will run now, but definitely not do what you want. You want the fifth column of df output, ie. the use percents. Why do you need -h/human readable output? We dont need that. Which row of df output do you want? I guess you don't want the header, ie. the first row: Filesystem 1K-blocks Used Available Use% Mounted on. Let's filter the columns with the disc name, I choose /dev/sda2. We can filter the row that has the first word equal to /dev/sda2 with grep "^/dev/sda2 ". The we need to get the value on the fifth column with awk '{print $5}'. We need to get rid of the '%' sign too, otherwise shell will not interpret the value as a number, with sed 's/%//' or better with tr -d '%'. Specifying date -d"today" is the same as just date. Enclosing a command in (...) runs it in subshell, we don't need that.
#!/bin/sh
temp=$(df | grep "^/dev/sda2 " | awk '{print $5}' | tr -d '%')
if [ "$temp" -gt 0 ]; then
echo "${temp}%"
date +"Date:%Y.%m.%d Hour: %H:%M"
fi
This is a simple, that if use percentage on disc /dev/sda2 is higher then 0, then it will print the use percentage and print current date and time in a custom format.
Assuming you're using GNU tools, you can narrow the df output to just what you need:
pct=$( df --output=pcent / | grep -o '[[:digit:]]\+' )
if [[ $pct -gt 60 ]]; then ...

Shell : What does this script do?

#!/bin/bash
if test $# -ne 2
then
echo "Error : Invalid number of arguments"
else
if [ -d $1 ]
then
if [[ $2 =~ ^[0-9]+$ ]]
then
ls -l $1 | while read line
do
eval "echo $line | cut -d' ' -f5" | while read ln
do
if [[ $ln -gt $2 ]]
then
echo $line
fi
done
done
else
echo $2" is not a integer"
fi
else
echo "The repertory "$1" does not exist "
fi
fi
The question was to make cpp , that works like the command cp . The script it's supposed to react correctly if we don't give 2 argument. I don't understand what this script do from line 10 .
This code is the following of this post Explain me 2 lines of this shell script.
Thanks
Without working through the code line by line and explaining it, I would point you at http://explainshell.com, which takes lines of shell code and puts commentary from the manual on each parameter.
E.g, this is part of line 12 above: http://explainshell.com/explain?cmd=echo+%24line+%7C+cut+-d%27+%27+-f5
It should help you go through it line by line and work out what is going on.
In words: It selects the lines from ls -l from a directory $1 which have a size bigger than $2.
If that code is in a file called script.sh, it is called like:
$ ./script.sh /home/user 130000
And it will print all lines of ls -l /home/user which have a size bigger than 130000.
I do not know why the eval in:
eval "echo $line | cut -d' ' -f5" | while read ln
The line will work the same as:
echo $line | cut -d' ' -f5 | while read ln

ksh: shell script to search for a string in all files present in a directory at a regular interval

I have a directory (output) in unix (SUN). There are two types of files created with timestamp prefix to the file name. These file are created on a regular interval of 10 minutes.
e. g:
1. 20140129_170343_fail.csv (some lines are there)
2. 20140129_170343_success.csv (some lines are there)
Now I have to search for a particular string in all the files present in the output directory and if the string is found in fail and success files, I have to count the number of lines present in those files and save the output to the cnt_succ and cnt_fail variables. If the string is not found I will search again in the same directory after a sleep timer of 20 seconds.
here is my code
#!/usr/bin/ksh
for i in 1 2
do
grep -l 0140127_123933_part_hg_log_status.csv /osp/local/var/log/tool2/final_logs/* >log_t.txt; ### log_t.txt will contain all the matching file list
while read line ### reading the log_t.txt
do
echo "$line has following count"
CNT=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT=`expr $CNT - 1`
echo $CNT
done <log_t.txt
if [ $CNT > 0 ]
then
exit
fi
echo "waiitng"
sleep 20
done
The problem I'm facing is, I'm not able to get the _success and _fail in file in line and and check their count
I'm not sure about ksh, but while ... do; ... done is notorious for running off with whatever variables you're using in bash. ksh might be similar.
If I've understand your question right, SunOS has grep, uniq and sort AFAIK, so a possible alternative might be...
First of all:
$ cat fail.txt
W34523TERG
ADFLKJ
W34523TERG
WER
ASDTQ34T
DBVSER6
W34523TERG
ASDTQ34T
DBVSER6
$ cat success.txt
abcde
defgh
234523452
vxczvzxc
jkl
vxczvzxc
asdf
234523452
vxczvzxc
dlkjhgl
jkl
wer
234523452
vxczvzxc
And now:
egrep "W34523TERG|ASDTQ34T" fail.txt | sort | uniq -c
2 ASDTQ34T
3 W34523TERG
egrep "234523452|vxczvzxc|jkl" success.txt | sort | uniq -c
3 234523452
2 jkl
4 vxczvzxc
Depending on the input data, you may want to see what options sort has on your system. Examining uniq's options may prove useful too (it can do more than just count duplicates).
Think you want something like this (will work in both bash and ksh)
#!/bin/ksh
while read -r file; do
lines=$(wc -l < "$file")
((sum+=$lines))
done < <(grep -Rl --include="[1|2]*_fail.csv" "somestring")
echo "$sum"
Note this will match files starting with 1 or 2 and ending in _fail.csv, not exactly clear if that's what you want or not.
e.g. Let's say I have two files, one starting with 1 (containing 4 lines) and one starting with 2 (containing 3 lines), both ending in `_fail.csv somewhere under my current working directory
> abovescript
7
Important to understand grep options here
-R, --dereference-recursive
Read all files under each directory, recursively. Follow all
symbolic links, unlike -r.
and
-l, --files-with-matches
Suppress normal output; instead print the name of each input
file from which output would normally have been printed. The
scanning will stop on the first match. (-l is specified by
POSIX.)
Finaly I'm able to find the solution. Here is the complete code:
#!/usr/bin/ksh
file_name="0140127_123933.csv"
for i in 1 2
do
grep -l $file_name /osp/local/var/log/tool2/final_logs/* >log_t.txt;
while read line
do
if [ $(echo "$line" |awk '/success/') ] ## will check the success file
then
CNT_SUCC=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT_SUCC=`expr $CNT_SUCC - 1`
fi
if [ $(echo "$line" |awk '/fail/') ] ## will check the fail file
then
CNT_FAIL=`wc -l $line|tr -s " "|cut -d" " -f2`
CNT_FAIL=`expr $CNT_FAIL - 1`
fi
done <log_t.txt
if [ $CNT_SUCC > 0 ] && [ $CNT_FAIL > 0 ]
then
echo " Fail count = $CNT_FAIL"
echo " Success count = $CNT_SUCC"
exit
fi
echo "waitng for next search..."
sleep 10
done
Thanks everyone for your help.
I don't think I'm getting it right, but You can't diffrinciate the files?
maybe try:
#...
CNT=`expr $CNT - 1`
if [ $(echo $line | grep -o "fail") ]
then
#do something with fail count
else
#do something with success count
fi

How to verify information using standard linux/unix filters?

I have the following data in a Tab delimited file:
_ DATA _
Col1 Col2 Col3 Col4 Col5
blah1 blah2 blah3 4 someotherText
blahA blahZ blahJ 2 someotherText1
blahB blahT blahT 7 someotherText2
blahC blahQ blahL 10 someotherText3
I want to make sure that the data in 4th column of this file is always an integer. I know how to do this in perl
Read each line, Store value of 4th column in a variable
check if that variable is an integer
if above is true, continue the loop
else break out of the loop with message saying file data not correct
But how would I do this in a shell script using standard linux/unix filter? My guess would be to use grep, but I am not sure how?
cut -f4 data | LANG=C grep -q '[^0-9]' && echo invalid
LANG=C for speed
-q to quit at first error in possible long file
If you need to strip the first line then use tail -n+2 or you could get hacky and use:
cut -f4 data | LANG=C sed -n '1b;/[^0-9]/{s/.*/invalid/p;q}'
awk is the tool most naturally suited for parsing by columns:
awk '{if ($4 !~ /^[0-9]+$/) { print "Error! Column 4 is not an integer:"; print $0; exit 1}}' data.txt
As you get more complex with your error detection, you'll probably want to put the awk script in a file and invoke it with awk -f verify.awk data.txt.
Edit: in the form you'd put into verify.awk:
{
if ($4 !~/^[0-9]+$/) {
print "Error! Column 4 is not an integer:"
print $0
exit 1
}
}
Note that I've made awk exit with a non-zero code, so that you can easily check it in your calling script with something like this in bash:
if awk -f verify.awk data.txt; then
# action for success
else
# action for failure
fi
You could use grep, but it doesn't inherently recognize columns. You'd be stuck writing patterns to match the columns.
awk is what you need.
I can't upvote yet, but I would upvote Jefromi's answer if I could.
Sometimes you need it BASH only, because tr, cut & awk behave differently on Linux/Solaris/Aix/BSD/etc:
while read a b c d e ; do [[ "$d" =~ ^[0-9] ]] || echo "$a: $d not a numer" ; done < data
Edited....
#!/bin/bash
isdigit ()
{
[ $# -eq 1 ] || return 0
case $1 in
*[!0-9]*|"") return 0;;
*) return 1;;
esac
}
while read line
do
col=($line)
digit=${col[3]}
if isdigit "$digit"
then
echo "err, no digit $digit"
else
echo "hey, we got a digit $digit"
fi
done
Use this in a script foo.sh and run it like ./foo.sh < data.txt
See tldp.org for more info
Pure Bash:
linenum=1; while read line; do field=($line); if ((linenum>1)); then [[ ! ${field[3]} =~ ^[[:digit:]]+$ ]] && echo "FAIL: line number: ${linenum}, value: '${field[3]}' is not an integer"; fi; ((linenum++)); done < data.txt
To stop at the first error, add a break:
linenum=1; while read line; do field=($line); if ((linenum>1)); then [[ ! ${field[3]} =~ ^[[:digit:]]+$ ]] && echo "FAIL: line number: ${linenum}, value: '${field[3]}' is not an integer" && break; fi; ((linenum++)); done < data.txt
cut -f 4 filename
will return the fourth field of each line to stdout.
Hopefully that's a good start, because it's been a long time since I had to do any major shell scripting.
Mind, this may well not be the most efficient compared to iterating through the file with something like perl.
tail +2 x.x | sort -n -k 4 | head -1 | cut -f 4 | egrep "^[0-9]+$"
if [ "$?" == "0" ]
then
echo "file is ok";
fi
tail +2 gives you all but the first line (since your sample has a header)
sort -n -k 4 sorts the file numerically on the 4th column, letters will rise to the top.
head -1 gives you the first line of the file
cut -f 4 gives you the 4th column, of the first line
egrep "^[0-9]+$" checks if the value is a number (integers in this case).
If egrep finds nothing, $? is 1, otherwise it's 0.
There's also:
if [ `tail +2 x.x | wc -l` == `tail +2 x.x | cut -f 4 | egrep "^[0-9]+$" | wc -l` ] then
echo "file is ok";
fi
This will be faster, requiring two simple scans through the file, but it's not a single pipeline.
#OP, use awk
awk '$4+0<=0{print "not ok";exit}' file

Resources