I have this scenarios:
File Content:
10.1.1.1
10.1.1.2
10.1.1.3
10.1.1.4
I want sed or awk so that when i cat the file every time new line is returned.
like
First iteration:
cat ip | some magic
10.1.1.1
Second iteration returns
10.1.1.2
Third iteration returns
10.1.1.3
Fourth iteration returns
10.1.1.4
and after n number of iterations, it returns to line 1
Fifth iteration returns:
10.1.1.1
Can we do it using sed or awk.
You will need to store the line number in a file and increment it with modulus at each invocation.
get_line () {
if [[ ! -e /var/local/get_line.next ]]
then
if [[ ! -e /var/local ]]
then
mkdir -p /var/local
fi
line_no=1
else
line_no=$(< /var/local/get_line.next)
fi
file_length=(wc -l < ip_file)
if ((file_length == 0))
then
echo "Error: Data file is empty" >&2
return 1
fi
if ((line > file_length))
then
line=1
fi
sed -n "$line_no{p;q}" ip_file
echo "$((++line_no))" > /var/local/get_line.next
}
This is in the form of a function which you can incorporate in a script. Feel free to change the location of the get_line.next file. Note that permissions will need to be correct to read or write the files or to create the directory, if necessary.
You will not need to use cat.
You can't do this with cat. You also can't seek on a pipe so you can't use a pipe ..
You can do this with a nested while loop
while ((1))
do
while read line
do
echo "$line"
done <somefile
done
Related
I'm trying to do the following:
I have this file called testing.txt that I want to update each time the ip address or addresses change based on the name (test1ip, test2ip):
127.0.0.1 localhost
somotherrandomip testing
192.168.0.36 test1ip
192.168.0.37 test2ip
This is what I've tried.
#!/bin/bash
array=(
"192.168.0.34 test1ip"
"192.168.0.35 test2ip"
)
for i in "${array[#]}"; do
if ! grep -Fxq "$i" testing.txt
then
echo "ip-name=$i is not present, so adding it in testing.txt file"
echo "$i" >> testing.txt
else
echo "ip-name=$i is present in file, so nothing to do"
fi
done
However, this script appends a completely new line if the line is not found. What I would like to achieve is to overwrite the line if test1ip or test2ip is found but the ip address change.
Expected result:
127.0.0.1 localhost
somotherrandomip testing
192.168.0.34 test1ip
192.168.0.35 test2ip
I've also read this How to check if a string contains a substring in Bash but it seems i can't figure it out.
Any help is greatly appreciated!
The following works on my machine. I changed the array to an associative array, removed the -x option to grep, and used sed to edit the file in place.
#!/bin/bash
#associative array
declare -A array=(
[test1ip]="192.168.0.34"
[test2ip]="192.168.0.35"
)
#Loop over keys of the array
#See parameter expansion in bash manpage
for i in "${!array[#]}"; do
if ! grep -Fq "$i" testing.txt
then
echo "ip-name=$i is not present, so adding it in testing.txt file"
echo "${array[$i]} $i" >> testing.txt
else
echo "ip-name=$i is present in file so running sed"
#Escape sed left hand side
#Adapted for extended regular expressions from https://unix.stackexchange.com/a/129063/309759
i=$(printf '%s\n' "$i" | sed 's:[][\\/.^$*+?(){}|]:\\&:g')
#Excape right hand side
ipaddr=$(printf '%s\n' "${array[$i]}" | sed 's:[\\/&]:\\&:g;$!s/$/\\/')
#Replace old IP vith new IP
sed -Ei "s/[0-9]+(\.[0-9]+){3} +$i/$ipaddr $i/" testing.txt
fi
done
Here's a bash + awk solution that'll do the job efficiently:
#!/bin/bash
array=(
"192.168.0.34 test1ip"
"192.168.0.35 test2ip"
"192.168.0.33 test3ip"
)
awk '
FNR == NR {
aarr[$2] = $1
next
}
! ($2 in aarr)
END {
for (host in aarr)
print aarr[host], host
}
' <(printf '%s\n' "${array[#]}") testing.txt
127.0.0.1 localhost
somotherrandomip testing
192.168.0.33 test3ip
192.168.0.34 test1ip
192.168.0.35 test2ip
notes:
bash's <( commands ) is called process-subtitution. It creates a file containing the output of commands that you can use as argument.
awk's FNR == NR is a condition for selecting the first file argument. In this block I create an associative array that translates a hostname to its new IP address.
! ($2 in aarr) means to print the records for which the hostname is not in the translation array.
The END is for printing the translation array (new IPs of hostnames).
I am learning bash. I would like to get the return value and matched line by grep at once.
if cat 'file' | grep 'match_word'; then
match_by_grep="$(cat 'file' | grep 'match_word')"
read a b <<< "${match_by_grep}"
fi
In the code above, I used grep twice. I cannot think of how to do it by grep once. I am not sure match_by_grep is always empty even when there is no matched words because cat may output error message.
match_by_grep="$(cat 'file' | grep 'match_word')"
if [[ -n ${match_by_grep} ]]; then
# match_by_grep may be an error message by cat.
# So following a and b may have wrong value.
read a b <<< "${match_by_grep}"
fi
Please tell me how to do it. Thank you very much.
You can avoid the double use of grep by storing the search output in a variable and seeing if it is not empty.
Your version of the script without double grep.
#!/bin/bash
grepOutput="$(grep 'match_word' file)"
if [ ! -z "$grepOutput" ]; then
read a b <<< "${grepOutput}"
fi
An optimization over the above script ( you can remove the temporary variable too)
#!/bin/bash
grepOutput="$(grep 'match_word' file)"
[[ ! -z "$grepOutput" ]] && (read a b <<< "${grepOutput}")
Using double-grep once for checking if-condition and once to parse the search result would be something like:-
#!/bin/bash
if grep -q 'match_word' file; then
grepOutput="$(grep 'match_word' file)"
read a b <<< "${grepOutput}"
fi
When assigning a variable with a string containing a command expansion, the return code is that of the (rightmost) command being expanded.
In other words, you can just use the assignment as the condition:
if grepOutput="$(cat 'file' | grep 'match_word')"
then
echo "There was a match"
read -r a b <<< "${grepOutput}"
(etc)
else
echo "No match"
fi
Is this what you want to achieve?
grep 'match_word' file ; echo $?
$? has a return value of the command run immediately before.
If you would like to keep track of the return value, it will be also useful to have PS1 set up with $?.
Ref: Bash Prompt with Last Exit Code
If I need to start grepping a file from line num 1293 all the way to the end of the file how can I do that?
More detailed info in case it helps:
I am trying to whip a quick function in my bashrc that lets me quickly search vim snippet files for a particular snippet echoing the snippet name and associated command(s) to screen. So I have no probs getting the line num for the snippet name and even printing out the command on the following line num. But if the snippet is a multi-line command then I need to grep for the next line beginning with snippet "^snippet " and then return all lines between, but I cannot find any details how I can go about getting grep to start its search starting from a particular line num.
A secondary question is how in a .bashrc function can I exit the function early? When I use the 'exit' command
currently commented out in the funct below the terminal itself exits/closes rather than just exiting the funct.
function vsls() {
if [[ "$2" =~ ^(html|sh|vim)$ ]] ; then
sPath="$2".snippets
elif [[ "$2" =~ ^(html|sh|vim).snippets$ ]] ; then
sPath="$2"
else
echo "\nExiting. You did not enter a recognized vim snippets file name."
# exit 69
fi
lnN=$(more $HOME/.vim/snippets/"$sPath"|grep -nm 1 $1|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$lnN: ${lnN}"
cntr="$lnN"
sed -n "$cntr"p "$HOME/.vim/snippets/$sPath"
((cntr++))
sed -n "$cntr"p "$HOME/.vim/snippets/$sPath"
}
#chepner
I don't know why (lack of know-how likely) but without specifying 'more' I get a permissions error:
03:43 ~ $ fLNum=$($HOME/.vim/snippets/"$sPath"|grep -nm 1 tdotti|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$fLNum: ${fLNum}"
bash: /home/user/.vim/snippets/html.snippets: Permission denied
$fLNum:
03:43 ~ $ fLNum=$(more $HOME/.vim/snippets/"$sPath"|grep -nm 1 tdotti|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$fLNum: ${fLNum}"
$fLNum: 1293
Now working as desired:
I stuck with sed since I feel most comfortable using sed. I have used the -n print opt before, but not too often so it totally escaped my mind to try something like that.
function vsls() {
if [[ "$2" =~ ^(html|sh|vim)$ ]] ; then
sPath="$2".snippets
elif [[ "$2" =~ ^(html|sh|vim).snippets$ ]] ; then
sPath="$2"
else
echo "\nExiting. You did not enter a recognized vim snippets file name."
# exit 69
fi
fLNum=$(more $HOME/.vim/snippets/"$sPath"|grep -nm 1 "snippet $1"|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$fLNum: ${fLNum}" #get line number of the snippet name searched, entered as input $1
((tLNum1 = fLNum+=1)) ; echo "\$tLNum1: ${tLNum1}" # tmpLineNum is next line num from which to start next grep search for lineNum of next snippet entry to determine where commands of desired snippet end
tLNum2=$(sed -n "${tLNum1},$ p" $HOME/.vim/snippets/"$sPath"|grep -nm 1 "snippet"|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$tLNum2: ${tLNum2}" #lineNum of next 'snippet entry'
let sLNum=tLNum2+fLNum sLNum-=1 ; let sLNum-=1 ; echo "\$sLNum: ${sLNum}" #tmpLineNum2 is not actual line num in file, but rather the number of lines since the start of the second search, that is necessarily somewhere within the file: so if second search begins on line 1294, for all intents and purpose actual line num 1294 is line 1 of the new (second) search; therefore I need to add the tLNum2 with fLNum to determine actual lineNum in the of the next snippet entry
echo ""
sed -n "${fLNum},${sLNum} p" "$HOME/.vim/snippets/$sPath"
echo ""
}
But it is curious why I needed to do:
let sLNum=tLNum2+fLNum sLNum-=1 ; let sLNum-=1
to get the correct line number of the second grep search. I only got lucky fooling around, b/c I would have thought:
let sLNum=tLNum2+fLNum sLNum-=1
or:
let sLNum=tLNum2+fLNum ; let sLNum-=1
should have done the trick; that is, secondLineNum = tmpLNum2 + firstLineNum and then secondLineNum - 1. But the result would never end up 1 less but always equal to tLNum+fLNum. It would be good to learn why that did not work as expected.
But its working. so thanks.
Or with sed like this:
sed -n "1293,$ p" yourfile | grep xyz
Or, if the line number is in a variable called line:
sed -n "${line},$ p" yourfile | grep xyz
Or, if you want your grep to find nothing in the first 1292 lines, but still report the correct line number if you are using grep -n, you can just get the (empty) hold buffer for grep to look at for lines 1 to 1292
sed "1,1292g" yourfile | grep -n xyz
awk is better suited for this
awk '/search_pattern/ && NR > 1292' filename
tail -n +1293 file | grep ....
I am currently trying to evaluate txt-files in a directory using bash. I want to know if the third line of the txt-file matches a certain string. The file starts with two empty lines, then the target string. I tested the following one liner:
if [[ $(head -n 3 a_txt_file.txt) == "target_string" ]]; then echo yes; else echo no; fi
I can imagine that since head -n 3 also prints out the two empty lines, I have to add them to the if condition. But "\n\ntarget_string" and "\n\ntarget_string\n" also don't work.
How would one do this correctly (And I guess it can be done more elegantly as well)?
Try this instead - it will print only the third line:
sed -n 3p file.txt
If you just need to remove the top two lines:
head -n 3 | tail -1
You'll want to use sed instead of head. This gets the third line, tests if it matches, and then you can do whatever you want with it if it does match.
if [[ $(sed '3q;d' test_text.txt ) == "target_string" ]]; then echo yes; else echo no; fi
Besides sed you can try awk to print 3rd line
awk 'NR==3'
A pure bash solution:
if { read; read; read line; } < test_text.txt
[[ $line = target_string ]]
then
echo yes
else
echo no
fi < test_text.txt
This takes advantage of the fact that the condition of the if statement can be a sequence of commands. First, read twice from the file to discard the empty lines; the third sets line to the 3rd line. After that, you can test it against the target string.
I have the code in my bash scripts that works unstable:
# check every line of check_list file presents in my_prog output
MY_LIST=`./my_prog`
for l in $(cat check_list); do
if ! echo -n "$MY_LIST" | grep -q -x "$l"; then
die "Bad line: '$l'"
fi
done
This piece of code of my huge scripting pool shows "Bad line: 'smthng'" with probability around 1/5000. I wasn't able to represent this event by the naked script but only in my huge scripting pool.
However this code seems to work very fine:
# check every line of check_list file presents in my_prog output
./my_prog > my_list
for l in $(cat check_list); do
if ! grep -q -x "$l" "my_list"; then
die "Bad line: '$l'"
fi
done
The reason why I don't like the second statement is that its use an intermediate file "my_list".
What could be a problem of unstable working of the first statement?
Instead of calling grep for every line in your check_list, you can run one awk program:
awk '
FILENAME == ARGV[1] {check_list[$0]; next}
$0 in check_list {
print "bad line: " $0
exit 1
}
' check_list <(./my_prog)
Or, see if there are any common lines between your program's output and your check_list:
common=$( comm -12 <(sort -u check_list) <(./my_prog | sort -u) )
if [ -n "$common" ]; then
echo "bad lines: "
echo "$common"
die
fi
I don't know what's wrong with the first version but you can easily eliminate the creation of a temporary file.
Note the you'll have to correct the logic, i did not really understand that, probably you'll want to update a variable in the inner loop and decide whether to die after the inner loop.
for i in $*; do
for l in $(cat check_list); do
if ! echo "$i" | grep -q -x "$l"; then
die "Bad line: '$i', '$l'"
fi
done
done | ./my_prog