Follow the details, the site is asking me to include some text because there is mostly code, so i type this sentence, but i think it is self explanatory
Sample log file :
jue 08 abr 2021 13:33:49 -03 : VA John Doe : PING google.com (17x.2xx.1x2.4x): 56 data bytes
--- google.com ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 40.462/50.166/62.318 ms
jue 08 abr 2021 13:35:35 -03 : VA John Doe : PING google.com (17x.2xx.1x2.4x): 56 data bytes
--- google.com ping statistics ---
100 packets transmitted, 99 packets received, 1% packet loss
round-trip min/avg/max = 42.055/48.856/136.962 ms
jue 08 abr 2021 13:37:21 -03 : VA John Doe : PING google.com (17x.2xx.1x2.4x): 56 data bytes
--- google.com ping statistics ---
100 packets transmitted, 100 packets received, 0% packet loss
round-trip min/avg/max = 40.058/47.762/64.169 ms
my command so far :
cat sample.log | sed -r -e '/^... [0-9]+ ... [0-9]{4} [0-9]{2}:[0-9]{2}/{s/(... [0-9]+ ... [0-9]{4} [0-9]{2}:[0-9]{2}).*$/\1/g;n;d}' -e '/^--- google.*$/d' -e 's/100 packets transmitted.*([0-9]+%) packet.*$/\1/' -e '/round-trip/d'
result obtained :
jue 08 abr 2021 13:33
0%
jue 08 abr 2021 13:35
1%
jue 08 abr 2021 13:37
0%
desired ideal result :
jue 08 abr 2021 13:33, 0%
jue 08 abr 2021 13:35, 1%
jue 08 abr 2021 13:37, 0%
Let's assume:
The packet loss percentage is always found in lines ending with NUM% packet loss.
The date and time are always found in lines ending with data bytes.
Then, with GNU sed (tested on the two complete records you show):
$ sed -nE '/packet loss$/{s/.*\s([0-9]+%) packet loss$/\1/;h}
/data bytes$/{s/(.{24}).*/\1/;G;s/\n/, /;p}' sample.log
jue 08 abr 2021 13:35:35, 0%
jue 08 abr 2021 13:37:21, 1%
1st solution: This should be a task for awk. With your shown samples, please try following awk code.
awk -v OFS=", " '
match($0,/^[a-zA-Z]+ [0-9]{2} [a-zA-Z]+ [0-9]{4} ([0-9]{2}:){2}[0-9]{2}/){
val=substr($0,RSTART,RLENGTH-3)
next
}
/packets transmitted/{
print val,$(NF-2)
val=""
}
' Input_file
Explanation: Simple explanation would be, using match function where mentioning regex to match ^[a-zA-Z]+ [0-9]{2} [a-zA-Z]+ [0-9]{4} ([0-9]{2}:){2}[0-9]{2}(explained regex in further), if a match of is found then creating val variable which has value of matched(caught value) by regex. Using next will skip all further statements from here. Then checking condition if line contains packets transmitted then print val along with 3rd last field of that line. Nullifying val variable then.
Explanation of regex:
^[a-zA-Z]+ ##Matching small/capital letters 1 or more occurrences from starting.
[0-9]{2} ##Matching space followed by 2 occurrences of digits.
[a-zA-Z]+ ##Matching space followed by 2 occurrences of small/capital letters.
[0-9]{4} ##Matching space followed by followed by 4 digits.
([0-9]{2}:){2}[0-9]{2} ##Matching space followed by digits 2 occurrences followed by colon and this whole group should occur 2 times followed by 2 occurrences of digits.
2nd solution: Using GNU awk here we can use almost same mentioned regex in RS variable and can get desired results as follows:
awk -v RS='[a-zA-Z]+ [0-9]{2} [a-zA-Z]+ [0-9]{4} [0-9]{2}:[0-9]{2}|[0-9]{1,3}%' -v OFS=", " '
RT{
val=(val?val (++count%2==0?ORS:OFS):"") RT
}
END{
print val
}
' Input_file
To get the desired format, you can pipe the output to:
sed 'N;s/\n/, /'
The final command becomes (note that you don't need to cat to sed as it accepts the filename as an argument):
sed -r -e '/^... [0-9]+ ... [0-9]{4} [0-9]{2}:[0-9]{2}/{s/(... [0-9]+ ... [0-9]{4} [0-9]{2}:[0-9]{2}).*$/\1/g;n;d}' -e '/^--- google.*$/d' -e 's/100 packets transmitted.*([0-9]+%) packet.*$/\1/' -e '/round-trip/d' sample.log | sed 'N;s/\n/, /'
Output:
jue 08 abr 2021 13:33, 0%
jue 08 abr 2021 13:35, 1%
jue 08 abr 2021 13:37, 0%
I am trying to capture the sum of input rate and output rate for interfaces which are part of GigabitEthernet102.. I have some logic but would like assistance converting it to a suitable program to achieve this.
INPUT
GigabitEthernet102/0/0/1 is up, line protocol is up
Interface state transitions: 5
Hardware is GigabitEthernet/IEEE 802.3 interface(s), address is sets.sets.sets
Description: xxx
Internet address is x.x.x.x
--More-- MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 100Mb/s, 1000BASE-T, link type is force-up
output flow control is off, input flow control is off
Carrier delay (up) is 100 msec, Carrier delay (down) is 100 msec
loopback not set,
Last link flapped 10w3d
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
5638310 packets input, 467600021 bytes, 27 total input drops
0 drops for unrecognized upper-level protocol
Received 4 broadcast packets, 186012 multicast packets
1 runts, 0 giants, 0 throttles, 0 parity
2 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
4336700 packets output, 306662485 bytes, 0 total output drops
Output 44 broadcast packets, 0 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
GigabitEthernet102/0/0/2 is administratively down, line protocol is administratively down
Interface state transitions: 0
Hardware is GigabitEthernet/IEEE 802.3 interface(s), address is sets.sets.sets
Internet address is Unknown
MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
--More-- Full-duplex, 100Mb/s, 1000BASE-T, link type is force-up
output flow control is off, input flow control is off
Carrier delay (up) is 100 msec, Carrier delay (down) is 100 msec
loopback not set,
Last input never, output never
Last clearing of "show interface" counters never
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes, 0 total input drops
0 drops for unrecognized upper-level protocol
Received 0 broadcast packets, 0 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
0 packets output, 0 bytes, 0 total output drops
Output 0 broadcast packets, 0 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
TenGigE0/2/0/3 is up, line protocol is up
Interface state transitions: 3
Hardware is TenGigE, address is sets.sets.sets
Layer 1 Transport Mode is LAN
--More-- Description:
Internet address is x.x.x.x
MTU 1536 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, LR, link type is force-up
output flow control is off, input flow control is off
Carrier delay (up) is 10 msec
loopback not set,
Last link flapped 18w3d
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters 10w1d
5 minute input rate 0 bits/sec, 1 packets/sec
5 minute output rate 0 bits/sec, 1 packets/sec
3907990 packets input, 341219535 bytes, 0 total input drops
10323 drops for unrecognized upper-level protocol
Received 0 broadcast packets, 218445 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
6795781 packets output, 490559886 bytes, 0 total output drops
Output 0 broadcast packets, 0 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
GigabitEthernet102/0/0/3 is up, line protocol is up
Interface state transitions: 1
Hardware is GigabitEthernet/IEEE 802.3 interface(s), address is x.x.x.x
Description: xxx
Internet address is x.x.x.x
MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)
reliability 255/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Full-duplex, 100Mb/s, 1000BASE-T, link type is force-up
output flow control is off, input flow control is off
Carrier delay (up) is 100 msec, Carrier delay (down) is 100 msec
loopback not set,
--More-- Last link flapped 18w3d
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 2000 bits/sec, 2 packets/sec
5 minute output rate 1000 bits/sec, 2 packets/sec
23543445 packets input, 2623268477 bytes, 66 total input drops
18599 drops for unrecognized upper-level protocol
Received 37519 broadcast packets, 225636 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
24806600 packets output, 2739397694 bytes, 0 total output drops
Output 3 broadcast packets, 0 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
SUGGESTED PROGRAM FLOW
1> Capture interfaces which are UP and have GigabitEthernet102/[0-9]{3} syntax
GigabitEthernet102/0/0/1
GigabitEthernet102/0/0/3
2> Search file again with these interfaces (possibly positive lookbehind?) to capture input rate. Do the same thing for output rate.
3> Use array to perform sum of the input/output rates.
DESIRED OUTPUT
Total input traffic: x bits
Total output traffic: x bits
ATTEMPTS
1> awk '/^GigabitEthernet102/ && !/down/' sat_int_acc.txt
GigabitEthernet102/0/0/1 is up, line protocol is up
GigabitEthernet102/0/0/3 is up, line protocol is up
GigabitEthernet102/0/0/4 is up, line protocol is up
...
2> awk '/^GigabitEthernet102/ && !/down/ {intf[$1]} END {for(i in intf) print i}' sat_int_acc.txt
GigabitEthernet102/0/0/30
GigabitEthernet102/0/0/21
GigabitEthernet102/0/0/12
GigabitEthernet102/0/0/40
...
awk '/^GigabitEthernet102/ && !/down/; /^GigabitEthernet102/,/input rate/' sat_int_acc.txt
Thanks.
The following does input and output rates, based on your attempt with regular expression ranges. It reads a block from a line starting with GigabitEthernet102 and containing is up until a blank line or the end of file. Within each block it looks for input rate or output rate and sums the rates.
$ cat sat_int_acc.awk
/^GigabitEthernet102.* is up/,/^$/ {
if (/input rate/) input_total += $5
if (/output rate/) output_total += $5
}
END {
print "Total input traffic: " input_total " bits"
print "Total output traffic: " output_total " bits"
}
$ awk -f sat_int_acc.awk sat_int_acc.txt
Total input traffic: 2000 bits
Total output traffic: 1000 bits
This should work with one awk command.
awk '/protocol is/ {f=0} /GigabitEthernet102.* is up/ {f=1} f && /5 minute/ {sum+=$5} END {print sum}' file
3000
/protocol is/ {f=0} Test if start of new interface block and set flag f to 0
/GigabitEthernet102.* is up/ {f=1} if correct interface is found set flag f to 1
f && /5 minute/ {sum+=$5} if flag f\ is true and line is5 minute`
END {print sum}' then print sum of bout input and output
NB you need to not mix 5 minute and 30 second input.
Looks like I might have found a solution, but it is not pure awk.
$ awk '/GigabitEthernet102.* is up/,/input rate/' sat_int_acc.txt | grep "input rate" | awk '{x+=$5}END {print x}'
43237000
I'm rather new to Linux and would like to use a shell script to count the number of times a specific string appears in concurrent lines of text.
For example, I have a log with data similar to this:
active node table^M
-------------------^M
pkey vlan master-s/n gateway-s/n gateway-prio if-name gateway name advertised ip
0x7fff 0 N/A 0xa0600 100 if0 DA2WIBL1-25-io 10.126.144.49
0x7fff 0 N/A 0xa0580 100 if0 DA2WIBL1-31-io 10.126.144.51
0x7fff 0 N/A 0xa0400 100 if0 DA2WIBL1-28-io 10.126.144.50
active node table
-------------------
I want to simply count the number of concurrent lines containing the string: '0x7fff', and write that to a file.
Does anyone have an idea of how to wisely approach this? I know some shell, expect and similar scripting languages.
If you mean "consecutive" by "concurrent", then you normally use uniq command to group consecutive lines, uniq -c to group and count them simultaneously:
cat logfile | awk '{print $1}' | uniq -c | grep '0x7fff' | awk '{print $1}'
Given content of logfile as:
0x7fff 0 N/A 0xa0600 100 if0 DA2WIBL1-25-io 10.126.144.49
0x7fff 0 N/A 0xa0580 100 if0 DA2WIBL1-31-io 10.126.144.51
0x7fff 0 N/A 0xa0400 100 if0 DA2WIBL1-28-io 10.126.144.50
0x8b5f 0 N/A 0xa0600 100 if0 DA2WIBL1-25-io 10.126.144.49
0x7fff 0 N/A 0xa0400 100 if0 DA2WIBL1-28-io 10.126.144.50
, above command will produce the following output:
3
1
grep "0x7fff" logfilename.log | wc -l
the above code could help finding the count of number of lines in which the pattern or the string valye "0x7fff" appears.
$ awk -v key='0x7fff' '{if($1==key) c++; else {print c; c=0}}
END {print c}' file
test using #gudok's sample with
3
1
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 6 years ago.
Improve this question
[root#test143 ~]# ping 8.8.8.8 | while read xx; do echo "$(date '+%Y-%m-%d %H:%M:%S'): $xx"; done
2016-06-30 15:51:41: PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
2016-06-30 15:51:41: 64 bytes from 8.8.8.8: icmp_seq=1 ttl=47 time=78.2 ms
2016-06-30 15:51:43: 64 bytes from 8.8.8.8: icmp_seq=3 ttl=47 time=78.2 ms
2016-06-30 15:51:44: 64 bytes from 8.8.8.8: icmp_seq=4 ttl=47 time=78.3 ms
...
2016-06-30 15:57:58: 64 bytes from 8.8.8.8: icmp_seq=300 ttl=47 time=78.4 ms
I want to monitor the network interrupt(centos 6.5).
As mentioned above,we lost icmp_seq=2,How to locate the gap.
Search Stack again...
Say you have an input file named gaps
more gaps
icmp_seq=1
icmp_seq=2
icmp_seq=3
icmp_seq=5
icmp_seq=8
icmp_seq=10
You can make use of the below AWK
awk -F "=" '$2!=p+1{print p+1"-"$2-1}{p=$2}' gaps
4-4
6-7
9-9
explanations
-F "=" use = as the field separator
$2 is the second column from current input line
p is the previous value of the last line
so ($2!=p+1) is a condition : if $2 is different than previous value +1, then :
this part is executed : {print p+1 "-" $2-1} : print previous value +1, the - character and second columns + 1
{p=$2} is executed for each lines : p is assigned to the current 2nd column
I was wondering how to extract only certain data from the output of a shell command in Applescript. I want to be able to only pass the IP address into the variable from a "ping -o" command like this:
do shell script "ping -o " & blockedURL
-- Set the IP to blockedIP --
set blockedIP to ..
but I receive this:
"PING example.com (192.0.43.10): 56 data bytes 64 bytes from
192.0.43.10: icmp_seq=0 ttl=239 time=101.587 ms
--- example.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev =
101.587/101.587/101.587/0.000 ms"
When I execute the ping command I receive a lot of data I dont need. Is there any way of being able to only recall the (192.0.43.10)?
set a to "PING example.com (192.0.43.10): 56 data bytes 64 bytes from 192.0.43.10: icmp_seq=0 ttl=239 time=101.587 ms
--- example.com ping statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 101.587/101.587/101.587/0.000 ms"
set text item delimiters to "("
set temp to text item 2 of a
set text item delimiters to ")"
set temp to first text item of temp
return temp
The above is a full applescript solution. You can also use the following to get the IP just using the shell ping -o www.google.com | cut -d'(' -f2|cut -d')' -f1 | head -n1 so in applescript it would look like this :
do shell script "ping -o " & blockedURL & " | cut -d'(' -f2 | cut -d')' -f1 | head -n1"