I have these diff results saved to a file:
bash-3.00$ cat /tmp/voo
18633a18634
> sashabrokerSTP
18634a18636
> sashatraderSTP
21545a21548
> yheemustr
I just really need the logins:
bash-3.00$ cat /tmp/voo | egrep ">|<"
> sashaSTP
> sasha
> yhee
bash-3.00$
But when I try to iterate through them and just print the names I get errors.
I just do not understand the fundamentals of using "if" with "while loops".
Ultimately, I want to use the while loop because I want to do something to the lines - and apparently while only loads one line into memory at a time, as opposed to the whole file at once.
bash-3.00$ while read line; do if [[ $line =~ "<" ]] ; then echo $line ; fi ; done < /tmp/voo
bash-3.00$
bash-3.00$
bash-3.00$ while read line; do if [[ egrep "<" $line ]] ; then echo $line ; fi ; done < /tmp/voo
bash: conditional binary operator expected
bash: syntax error near `"<"'
bash-3.00$
bash-3.00$ while read line; do if [[ egrep ">|<" $line ]] ; then echo $line ; fi ; done < /tmp/voo
bash: conditional binary operator expected
bash: syntax error near `|<"'
bash-3.00$
There has to be a way to loop through the file and then do something to each line. Like this:
bash-3.00$ while read line; do if [[ $line =~ ">" ]];
then echo $line | tr ">" "+" ;
if [[ $line =~ "<" ]];
then echo $line | tr "<" "-" ;
fi ;
fi ;
done < /tmp/voo
+ sashab
+ sashat
+ yhee
bash-3.00$
You should be checking for >, not <, no?
while read line; do
if [[ $line =~ ">" ]]; then
echo $line
fi
done < /tmp/voo
Do you really need regex here? The following shell glob can also work:
while read line; do [[ "$line" == ">"* ]] && echo "$line"; done < /tmp/voo
OR use AWK:
awk '/^>/ { print "processing: " $0 }' /tmp/voo
grep will do:
$ grep -oP '> \K\w+' <<END
18633a18634
> sashabrokerSTP
18634a18636
> sashatraderSTP
21545a21548
> yheemustr
END
sashabrokerSTP
sashatraderSTP
yheemustr
Related
I have a text file which is the output difference of two grepped files . the text file has lines like below I need to read the file (loop through the lines in the text file ) and based on text to the left hand side of '<' and right hand side of '>' do something.
editing to add details:
LHS of < OR RHS of >
if either of those, i will need to store the content into a variable, and get the 1st(ABCDEF) 3rd(10) and search (will grep) for them in one of other two files and if found print a message and attach those file(s) names in an email DL. All the file names and directories have been stored in separate variables.
how do i do that.
ps:have basic knowledge on text formatting and bash/shell commands but still learning the scripting syntax.Thanks.
ABCDEF,20200101,10 <
PQRSTU,20200106,11 <
LMNOPQ,20200101,12 <
EFGHIJ,20200102,13 <
KLMNOP,20200103,14 <
STUVWX,20200104,15 <
PQRSTU,20200105,16 <
> LMNOPQ,20200101,10
ABCDEF,20200107,17 <
What wrong am I doing now?
while IFS= read -r line; do
if $line =~ ([^[:blank:]]+)[[:blank:]]+\<
then
IFS=, read -r f1 f2 f3 <<< "${BASH_REMATCH[1]}"
#echo "f1=$f1 f2=$f2 f3=$f3"
zgrep "$f1" file1 | grep "with seq $f3" || zgrep "$f1" file2 | grep "with seq $f3"
elif $line =~ \>[[:blank:]]+([^[:blank:]]+)
then
IFS=, read -r g1 g2 g3 <<< "${BASH_REMATCH[1]}"
#echo "g1=$g1 g2=$g2 g3=$g3"
zgrep "$g1" file3 | grep "with seq $g3" || zgrep "$g1" file3 | grep "with seq $g3"
fi
Would you please try something like:
#!/bin/bash
while IFS= read -r line; do
if [[ $line =~ ([^[:blank:]]+)[[:blank:]]+\< || $line =~ \>[[:blank:]]+([^[:blank:]]+) ]]; then
IFS=, read -r f1 f2 f3 <<< "${BASH_REMATCH[1]}"
echo "f1=$f1 f2=$f2 f3=$f3"
# do something here with "$f1", "$f2" and "$f3"
fi
done < file.txt
Output:
f1=ABCDEF f2=20200101 f3=10
f1=PQRSTU f2=20200106 f3=11
f1=LMNOPQ f2=20200101 f3=12
f1=EFGHIJ f2=20200102 f3=13
f1=KLMNOP f2=20200103 f3=14
f1=STUVWX f2=20200104 f3=15
f1=PQRSTU f2=20200105 f3=16
f1=LMNOPQ f2=20200101 f3=10
f1=ABCDEF f2=20200107 f3=17
Please modify the echo "f1=$f1 f2=$f2 f3=$f3" line to your desired
command such as grep.
The regex ([^[:blank:]]+)[[:blank:]]+\< matches a line which contains <
and assigns the bash variable ${BASH_REMATCH[1]} to the LHS.
On the other hand, the regex \>[[:blank:]]+([^[:blank:]]+) does the similar thing for
a line which contains >.
The statement IFS=, read -r f1 f2 f3 <<< "${BASH_REMATCH[1]}" splits the bash variable
on , and assigns f1, f2 and f3 to the fields.
Please note if the input file is very large, bash solution may not
be efficient in execution time. I used bash just because it will be convenient
to pass the variables to your grep command.
EDIT
Regarding the updated script in your question, please refer to the following modification:
while IFS= read -r line; do
if [[ $line =~ ([^[:blank:]]+)[[:blank:]]+\< ]]; then
IFS=, read -r f1 f2 f3 <<< "${BASH_REMATCH[1]}"
# echo "f1=$f1 f2=$f2 f3=$f3"
result=$(zgrep "$f1" file1 | grep "with seq $f3" || zgrep "$f1" file2 | grep "with seq $f3")
elif [[ $line =~ \>[[:blank:]]+([^[:blank:]]+) ]]; then
IFS=, read -r g1 g2 g3 <<< "${BASH_REMATCH[1]}"
# echo "g1=$g1 g2=$g2 g3=$g3"
result=$(zgrep "$g1" file3 | grep "with seq $g3" || zgrep "$g1" file3 | grep "with seq $g3")
fi
if [[ -n $result ]]; then
echo "result = $result"
fi
done < file.txt
i have a created a code to parse hostname,username and privilege from files in a dir..how ever i am not able to exit at the end of the code .the loop gives perfact data output.but not giving shell prompt back.
array arg[] contain name of file value
c=0
while((c<=${#arg[*]}))
do
hname=`grep -e "^hostname" ${arg[$c]} |awk '{ print $2 }'| sort |uniq`
if [[ -n $hname ]]
then
logDebug "data found for the $hname"
while read -r line; do
for term in "echo $line"; do
if [[ "$term" =~ (username)[[:space:]](.*?) ]]; then
userid=`echo "${BASH_REMATCH[2]}" | awk '{print $1}'`
logDebug "username: $userid"
if [[ "$term" =~ (privilege)[[:space:]](.*?) ]]; then
Priv=`echo "${BASH_REMATCH[2]}" | awk '{print $1}'`
PRIV=`echo "level=$Priv"`
logDebug "Privilege: $PRIV"
echo"$CUSTOMER|S|$hname|$OS|$userid|$Uid_Conv|$Uic_mode|$State|$i_login|$group|$PRIV|" >> "$OutputFile"
fi
fi
done
done < "${arg[$c]}"
fi
let c=c+1
done
For your first while clause...I would use
for ((c=0;c<${#arg[*];c++)); do
.....
done
Getting rid of the c=0 and let c=c+1
so in a bash script, have a while loop that read lines from the outpt of some pippelined sort commands. i get an error: line 13: syntax error near unexpected token 'done'
line 13 is the last line, where the "done" and the pipeline is. my script:
#!/bin/bash
while read -a line; do
if [[ ${line[2]} < $1 ]]; then
continue
fi
if [[ $2 != -MM ]]; then
if [[ ${line[3]} = N ]]; then
continue
fi
fi
echo -n "${line[0]} "
echo ${line[1]}
done < <(sort -b rooms | sort -sk2 | sort -sk4 | sort -sk3)
tnx.
I would first try to write the date to temp so I can see what the sorting is doing by inspecting the temp file. Then read that in line by line.
#!/usr/bin/env bash
TMPF="/tmp/tmp-file.dat"
sort -b rooms | sort -sk2 | sort -sk4 | sort -sk3 > "${TMPF}"
while IFS= read -r line ;do
if [[ ${line[2]} -ge $1 ]] && [[ $2 != -MM ]] && [[ ${line[3]} = N ]]; then
echo -n "${line[0]} "
echo ${line[1]}
fi
done < "${TMPF}"
You can use without using ()
like done < rooms |sort -b | sort -sk2 | sort -sk4 | sort -sk3
I would like to parse each line of my file that looks like this:
www.hello.by
https://www.google.com
http://www.stackoverflow.com
www.fedia.com
I want my script append http:// to each line if it doesn't start with http:// or https://
Did you even try to implement that? It's super simple..
You just need to read the file line by line. You can do this like that:
while read line ; do
echo $line
done < your_file
Then you have to check if your line starts with http:// or https://:
if [[ $line =~ ^https?:// ]] ; then
echo $line
fi
..and add http if it does not:
if [[ $line =~ ^https?:// ]] ; then
echo $line
else
echo http://$line
fi
Putting it all together you get:
while read line ; do
if [[ $line =~ ^https?:// ]] ; then
echo $line
else
echo http://$line
fi
done < your_file
You may want to save changes to a file, then redirect the output to a temporary file first and replace existing file with it at the end:
while read line ; do
if [[ $line =~ ^https?:// ]] ; then
echo $line
else
echo http://$line
fi
done < data > data.tmp
mv data{.tmp,}
Alternative awk version:
awk '{if($0 ~ /^https?/) print; else print "http://"$0}' data
Modify the file:
awk '{if($0 ~ /^https?/) print; else print "http://"$0}' your_file > your_file.tmp ; mv your_file{.tmp,}
Using sed
sed '\|^https\{0,1\}://|!s,^,http://,' file
http://www.hello.by
https://www.google.com
http://www.stackoverflow.com
http://www.fedia.com
Try the following sed command
sed -i '/https\?:\/\//{t;}
s/^/http:\/\//
' YOUR_FILE
I'm trying to put together a bash script that will search a bunch of files and if it finds a particular string in a file, it will add a new line on the line after that string and then move on to the next file.
#! /bin/bash
echo "Creating variables"
SEARCHDIR=testfile
LINENUM=1
find $SEARCHDIR* -type f -name *.xml | while read i; do
echo "Checking $i"
ISBE=`cat $i | grep STRING_TO_SEARCH_FOR`
if [[ $ISBE =~ "STRING_TO_SEARCH_FOR" ]] ; then
echo "found $i"
cat $i | while read LINE; do
((LINENUM=LINENUM+1))
if [[ $LINE == "<STRING_TO_SEARCH_FOR>" ]] ; then
echo "editing $i"
awk -v "n=$LINENUM" -v "s=new line to insert" '(NR==n) { print s } 1' $i
fi
done
fi
LINENUM=1
done
the bit I'm having trouble with is
awk -v "n=$LINENUM" -v "s=new line to insert" '(NR==n) { print s } 1' $i
if I just use $i at the end, it will output the content to the screen, if I use $i > $i then it will just erase the file and if I use $i >> $i it will get stuck in a loop until the disk fills up.
any suggestions?
Unfortunately awk dosen't have an in-place replacement option, similar to sed's -i, so you can create a temp file and then remove it:
awk '{commands}' file > tmpfile && mv tmpfile file
or if you have GNU awk 4.1.0 or newer, the -i inplace is added, so you can do:
awk -i inplace '{commands}' file
to modify the original
#cat $i | while read LINE; do
# ((LINENUM=LINENUM+1))
# if [[ $LINE == "<STRING_TO_SEARCH_FOR>" ]] ; then
# echo "editing $i"
# awk -v "n=$LINENUM" -v "s=new line to insert" '(NR==n) { print s } 1' $i
# fi
# done
# replaced by
sed -i 's/STRING_TO_SEARCH_FOR/&\n/g' ${i}
or use awk in place of sed
also
# ISBE=`cat $i | grep STRING_TO_SEARCH_FOR`
# if [[ $ISBE =~ "STRING_TO_SEARCH_FOR" ]] ; then
#by
if [ $( grep -c 'STRING_TO_SEARCH_FOR' ${i} ) -gt 0 ]; then
# if file are huge, if not directly used sed on it, it will be faster (but no echo about finding the file)
If you can, maybe use a temporary file?
~$ awk ... $i > tmpfile
~$ mv tmpfile $i
Or simply awk ... $i > tmpfile && mv tmpfile $i
Note that, you can use mktemp to create this temporary file.
Otherwise, with sed you can insert a line right after a match:
~$ cat f
auie
nrst
abcd
efgh
1234
~$ sed '/abcd/{a\
new_line
}' f
auie
nrst
abcd
new_line
efgh
1234
The command search if the line matches /abcd/, if so, it will append (a\) the line new_line.
And since sed as the -i to replace inline, you can do:
if [[ $ISBE =~ "STRING_TO_SEARCH_FOR" ]] ; then
echo "found $i"
echo "editing $i"
sed -i "/STRING_TO_SEARCH_FOR/{a
\new line to insert
}" $i
fi