Bash 4.3
Ubuntu 16.04
Each while read loop takes me a little under a second to accomplish. How can I grep for 3 results at the same time?
#!/bin/bash
#-- tmp files
tmp_dir="$(mktemp -d -t 'text.XXXXX' || mktemp -d 2>/dev/null)"
tmp_input1="${tmp_dir}/temp_input1.txt"
tmp_input2="${tmp_dir}/temp_input2.txt"
wDir="/home/work"
list="${wDir}/.ip-list.txt"
finalResults="${wDir}/final-results.txt"
cd "$wDir"
awk '{ print $11 }' "$list" | sort -u > "$tmp_input1"
while read ip; do
echo "-- IP Address: $ip" >> "$tmp_input2"
whois "$ip" | grep inetnum >> "$tmp_input2"
whois "$ip" | grep route >> "$tmp_input2"
whois "$ip" | grep mnt-by | head -n 2 | sed -n '1!p' >> "$tmp_input2"
echo "" >> "$tmp_input2"
done<"$tmp_input1"
mv "$tmp_input2" "$finalResults"
cat "$finalResults"
rm -rf "$tmp_dir"
Here is my .ip-list.txt file
> Tue Oct 16 21:15:59 2018 TCP 147.135.23.98 80 => 95.217.197.238 62293
> Tue Oct 16 21:16:52 2018 TCP 147.135.23.98 1160 => 95.217.243.116 44076
> Tue Oct 16 21:16:51 2018 TCP 147.135.23.98 1160 => 159.69.253.26 43842
> Tue Oct 16 21:16:47 2018 TCP 147.135.23.98 1160 => 95.217.49.21 13288
> Tue Oct 16 21:16:18 2018 TCP 147.135.23.98 80 => 95.217.223.72 21969
> Tue Oct 16 21:16:42 2018 TCP 147.135.23.98 1160 => 95.216.232.46 9834
> Tue Oct 16 21:16:54 2018 TCP 147.135.23.98 1160 => 88.198.149.27 23388
> Tue Oct 16 21:15:57 2018 TCP 147.135.23.98 80 => 95.217.72.11 38498
> Tue Oct 16 21:16:41 2018 TCP 147.135.23.98 1160 => 159.69.250.160 8549
> Tue Oct 16 21:16:27 2018 TCP 147.135.23.98 80 => 95.217.57.97 52546
> Tue Oct 16 21:16:28 2018 TCP 147.135.23.98 80 => 95.216.225.43 60635
> Tue Oct 16 21:16:32 2018 TCP 147.135.23.98 80 => 213.239.244.5 17729
> Tue Oct 16 21:16:05 2018 TCP 147.135.23.98 80 => 95.217.27.233 24669
> Tue Oct 16 21:16:46 2018 TCP 147.135.23.98 1160 => 94.130.60.83 21203
> Tue Oct 16 21:16:52 2018 TCP 147.135.23.98 1160 => 95.217.191.48 1070
> Tue Oct 16 21:16:22 2018 TCP 147.135.23.98 80 => 95.217.219.152 15617
> Tue Oct 16 21:16:44 2018 TCP 147.135.23.98 1160 => 95.217.35.111 55808
> Tue Oct 16 21:16:46 2018 TCP 147.135.23.98 1160 => 95.216.224.158 37768
> Tue Oct 16 21:16:13 2018 TCP 147.135.23.98 80 => 159.69.241.84 24365
> Tue Oct 16 21:16:21 2018 TCP 147.135.23.98 80 => 95.217.169.49 33710
> Tue Oct 16 21:16:07 2018 TCP 147.135.23.98 80 => 95.217.186.121 21758
> Tue Oct 16 21:16:00 2018 TCP 147.135.23.98 80 => 78.47.228.239 21199
> Tue Oct 16 21:16:30 2018 TCP 147.135.23.98 80 => 95.217.23.171 8670
> Tue Oct 16 21:16:49 2018 TCP 147.135.23.98 1160 => 95.216.244.96 22087
> Tue Oct 16 21:16:20 2018 TCP 147.135.23.98 80 => 95.217.64.54 13638
> Tue Oct 16 21:16:40 2018 TCP 147.135.23.98 1160 => 95.217.55.104 3377
> Tue Oct 16 21:16:09 2018 TCP 147.135.23.98 80 => 95.217.242.169 13627
> Tue Oct 16 21:16:54 2018 TCP 147.135.23.98 1160 => 95.217.192.169 6566
> Tue Oct 16 21:16:53 2018 TCP 147.135.23.98 1160 => 95.217.101.221 41547
> Tue Oct 16 21:16:54 2018 TCP 147.135.23.98 1160 => 159.69.227.235 62092
> Tue Oct 16 21:16:45 2018 TCP 147.135.23.98 1160 => 95.217.235.228 63643
> Tue Oct 16 21:16:08 2018 TCP 147.135.23.98 80 => 95.216.227.162 51332
> Tue Oct 16 21:16:54 2018 TCP 147.135.23.98 1160 => 95.217.68.128 38480
There are hundreds of lines.
How can I make these commands more efficient? Can they be combined?
whois "$ip" | grep inetnum >> "$tmp_input2"
whois "$ip" | grep route >> "$tmp_input2"
whois "$ip" | grep mnt-by | head -n 2 | sed -n '1!p' >> "$tmp_input2"
Write output of whois "$ip" to a variable and use variable:
grep -e 'inetnum' -e 'route' <<< "$out" >> "$tmp_input2"
grep 'mnt-by' <<< "$out" | sed '2!d' >> "$tmp_input2"
Not in this way.
The first two greps, you can replace by
whois "$ip" | egrep 'inetnum|route' >> "$tmp_input2"
But because you put the third grep is put through additional filters, you cannot add that one to the egrep.
But grep is not the problem; whois is the big time consumer. And you run it multiple times.
So, it would be a good idea to limit the number of whois-es.
hop=$(mktemp)
while read ip; do
echo "-- IP Address: $ip" >> "$tmp_input2"
whois "$ip" > $hop
grep inetnum $hop >> "$tmp_input2"
grep route $hop >> "$tmp_input2"
grep mnt-by $hop | head -n 2 | sed -n '1!p' >> "$tmp_input2"
echo "" >> "$tmp_input2"
done<"$tmp_input1
rm -f $hop
Related
I have a log file in which new lines are continuously written.
I would like a bash script that continuously reads the last line of this log file, so that I can process the line (e.g. execute a specific command if the line contains the word "error").
I've tried:
while true
do
if tail -n1 -f file.log | grep -q ERROR
then
echo "$(date) : ERROR detected"
fi
done
But it's spamming:
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
sun 21 mar 2021 18:32:41 CET : ERROR detected
(a new line is added every minute in this example)
How can I read only the last line and do not have spam for the result ?
I suggest with GNU grep:
tail -n1 -f file.log | grep --line-buffered ERROR | while read; do echo "$(date) : ERROR detected"; done
This is exactly why tail -f has been invented:
tail -f <logfile>
will show the last line of your logfile, so you can follow what gets added.
This can be combined with a grep:
tail -f <logfile> | grep <text_to_be_searched>
In your case:
tail -f file.log | grep "ERROR"
I need to parse all a file into a better format to produce an outcome with columns delimited by a comma, thinking of being able to export the content in CSV file.
This is an example of my input;
. D 0 Mon Dec 10 11:07:46 2018
.. D 0 Mon Feb 19 11:38:06 2018
RJ9-5 D 0 Fri Nov 30 10:34:24 2018
WorkingOnClass D 0 Wed Feb 28 09:37:52 2018
ML-Test001 D 0 Fri Dec 7 16:38:56 2018
TestML4Testing D 0 Wed Aug 22 08:58:42 2018
ML-NewDataSE SetCases1.xlsx A 1415577 Wed Aug 29 14:00:16 2018
DR0001-Dum01 D 0 Thu Aug 16 08:24:25 2018
DR0002-Dum02 D 0 Thu Aug 16 09:04:50 2018
Readme File for Documentation And Data Description.docx A 16136 Wed Aug 29 14:00:24 2018
ML Database Prototype D 0 Thu Dec 6 15:11:11 2018
OneNote D 0 Mon Dec 3 09:39:20 2018
Data A 0 Mon Dec 10 11:07:46 2018
\RJ9-5
. D 0 Fri Nov 30 10:34:24 2018
.. D 0 Mon Dec 10 11:07:46 2018
KLR0151_Set023_Files_RJ9_05.xlsx A 182462 Wed Apr 4 02:48:55 2018
KLR0152_Set023_Files_RJ9_05.xlsx A 525309 Wed Apr 4 02:53:57 2018
\ML-Test001
. D 0 Wed Feb 28 09:37:52 2018
.. D 0 Mon Dec 10 11:07:46 2018
WT_Conforming_Format1_1.docx A 500914 Mon Feb 26 08:50:55 2018
Conforming_Format_1_1.xlsx A 130647 Mon Feb 26 08:52:33 2018
DR0135_Dum01_text.xls A 974848 Mon Feb 12 08:11:11 2018
DR0139_Dum02_body.xls A 1061888 Tue Jun 19 13:43:54 2018
DataSet_File_mod0874953.xlsx A 149835 Mon Feb 26 14:17:02 2018
File Path For Dataset-2018.07.11.xlsx A 34661 Mon Feb 12 09:27:17
This is script right here can make the job:
#!/bin/bash
awk -v OFS=, '
BEGIN { print "PATH, FILENAME, SIZE, TIMESTAMP" }
/[\\]/ { path=$0 }
$2 ~ /A/ {print path"\\"$1,$3,$4 " " $5 " " $6 " " $7 " "$8 }
' "$#"
But is ignoring the names with spaces on it, so I need to validate them with something like:
awk -v FS="\t" '{print $1}'
But I could't integrate into the shell script, because the way the shell script is working, so I was thinking on make AWK to start reading by the end, since the end is always the same, and leave the rest.
The output should something like this:
/RJ9-5/KLR0151_Set023_Files_RJ9_05.xlsx,182462,Wed Apr 4 02:48:55 2018
/RJ9-5/KLR0152_Set023_Files_RJ9_05.xlsx,25309,Wed Apr 4 02:53:57 2018
/ML-Test001/WT_Conforming_Format1_1.docx,500914,Mon Feb 26 08:50:55 2018
/ML-Test001/Format_1_1.xlsx,130647,Mon Feb 26 08:52:33 2018
/ML-Test001/DR0135_Dum01_text.xls,974848,Mon Feb 12 08:11:11 2018
/ML-Test001/DR0139_Dum02_body.xls,1061888,Tue Jun 19 13:43:54 2018
/ML-Test001/DataSet_File_mod0874953.xlsx,149835,Mon Feb 26 14:17:02 2018
/ML-Test001/File Path For Dataset-2018.07.11.xlsx,34661,Mon Feb 12 09:27:17 2018
With GNU awk for the 3rd arg to match() (and far less importantly \s shorthand for [[:space:]]):
$ cat tst.awk
BEGIN { OFS="," }
{ gsub(/^\s+|\s+$/,"") }
sub(/^\\/,"/") { path = $0; next }
path == "" { next }
match($0,/^(.*[^ ]) +A +([^ ]+) +(.*)/,a) { print path "/" a[1], a[2], a[3] }
$ awk -f tst.awk file
/RJ9-5/KLR0151_Set023_Files_RJ9_05.xlsx,182462,Wed Apr 4 02:48:55 2018
/RJ9-5/KLR0152_Set023_Files_RJ9_05.xlsx,525309,Wed Apr 4 02:53:57 2018
/ML-Test001/WT_Conforming_Format1_1.docx,500914,Mon Feb 26 08:50:55 2018
/ML-Test001/Conforming_Format_1_1.xlsx,130647,Mon Feb 26 08:52:33 2018
/ML-Test001/DR0135_Dum01_text.xls,974848,Mon Feb 12 08:11:11 2018
/ML-Test001/DR0139_Dum02_body.xls,1061888,Tue Jun 19 13:43:54 2018
/ML-Test001/DataSet_File_mod0874953.xlsx,149835,Mon Feb 26 14:17:02 2018
/ML-Test001/File Path For Dataset-2018.07.11.xlsx,34661,Mon Feb 12 09:27:17
Try this Perl solution:
$ perl -lane ' if(/^\s*$/) { $x=0;$y=0} if(/^\\/) {$x=1 ;($a=$_)=~s/\s*$//g;$a=~s/\\/\//g; } $y++ if $x==1 ; if($y>3) { s/^\s*//g; $_=~s/(.+?)\s+\S+\s+((\d+)\s+.+)/$1 $2/g;print "$a/$_" } ' essparaq.txt
/RJ9-5/KLR0151_Set023_Files_RJ9_05.xlsx 182462 Wed Apr 4 02:48:55 2018
/RJ9-5/KLR0152_Set023_Files_RJ9_05.xlsx 525309 Wed Apr 4 02:53:57 2018
/ML-Test001/WT_Conforming_Format1_1.docx 500914 Mon Feb 26 08:50:55 2018
/ML-Test001/Conforming_Format_1_1.xlsx 130647 Mon Feb 26 08:52:33 2018
/ML-Test001/DR0135_Dum01_text.xls 974848 Mon Feb 12 08:11:11 2018
/ML-Test001/DR0139_Dum02_body.xls 1061888 Tue Jun 19 13:43:54 2018
/ML-Test001/DataSet_File_mod0874953.xlsx 149835 Mon Feb 26 14:17:02 2018
/ML-Test001/File Path For Dataset-2018.07.11.xlsx 34661 Mon Feb 12 09:27:17
$ cat essparaq.txt
. D 0 Mon Dec 10 11:07:46 2018
.. D 0 Mon Feb 19 11:38:06 2018
RJ9-5 D 0 Fri Nov 30 10:34:24 2018
WorkingOnClass D 0 Wed Feb 28 09:37:52 2018
ML-Test001 D 0 Fri Dec 7 16:38:56 2018
TestML4Testing D 0 Wed Aug 22 08:58:42 2018
ML-NewDataSE SetCases1.xlsx A 1415577 Wed Aug 29 14:00:16 2018
DR0001-Dum01 D 0 Thu Aug 16 08:24:25 2018
DR0002-Dum02 D 0 Thu Aug 16 09:04:50 2018
Readme File for Documentation And Data Description.docx A 16136 Wed Aug 29 14 :00:24 2018
ML Database Prototype D 0 Thu Dec 6 15:11:11 2018
OneNote D 0 Mon Dec 3 09:39:20 2018
Data A 0 Mon Dec 10 11:07:46 2018
\RJ9-5
. D 0 Fri Nov 30 10:34:24 2018
.. D 0 Mon Dec 10 11:07:46 2018
KLR0151_Set023_Files_RJ9_05.xlsx A 182462 Wed Apr 4 02:48:55 2018
KLR0152_Set023_Files_RJ9_05.xlsx A 525309 Wed Apr 4 02:53:57 2018
\ML-Test001
. D 0 Wed Feb 28 09:37:52 2018
.. D 0 Mon Dec 10 11:07:46 2018
WT_Conforming_Format1_1.docx A 500914 Mon Feb 26 08:50:55 2018
Conforming_Format_1_1.xlsx A 130647 Mon Feb 26 08:52:33 2018
DR0135_Dum01_text.xls A 974848 Mon Feb 12 08:11:11 2018
DR0139_Dum02_body.xls A 1061888 Tue Jun 19 13:43:54 2018
DataSet_File_mod0874953.xlsx A 149835 Mon Feb 26 14:17:02 2018
File Path For Dataset-2018.07.11.xlsx A 34661 Mon Feb 12 09:27:17
I have a file with data something like this :
{
MAG 121/002
Wed Mar 14 00:00:00 2018
MAG 121/003
Wed Mar 14 00:00:00 2018
MAG 121/004
Wed Mar 14 00:00:00 2018
}
I want the output as :
{
MAG 121/002 | Wed Mar 14 00:00:00 2018
MAG 121/003 | Wed Mar 14 00:00:00 2018
}
and so on.. Any help is appreciated.
What I tried was:
cat <filename> | awk '{printf "%s" (NR%2==0? RS:FS), $1}'
Could you please try following and let me know if this helps.
awk '/{/||/}/{print;next} /MAG/{val=$0;getline;print val OFS $0}' OFS=" | " Input_file
Solution with sed:
echo "MAG 121/002
Wed Mar 14 00:00:00 2018
MAG 121/003
Wed Mar 14 00:00:00 2018
MAG 121/004
Wed Mar 14 00:00:00 2018" | tr "\n" "|" | sed 's/|/ | /g' | sed -r 's/([^|]+\|[^|]+)\| /\1\n/g'
MAG 121/002 | Wed Mar 14 00:00:00 2018
MAG 121/003 | Wed Mar 14 00:00:00 2018
MAG 121/004 | Wed Mar 14 00:00:00 2018
Read and echo:
echo "MAG 121/002
Wed Mar 14 00:00:00 2018
MAG 121/003
Wed Mar 14 00:00:00 2018
MAG 121/004
Wed Mar 14 00:00:00 2018" | while read line ; do case $line in MAG*) echo -n $line "| " ;; *) echo $line ;; esac ; done
MAG 121/002 | Wed Mar 14 00:00:00 2018
MAG 121/003 | Wed Mar 14 00:00:00 2018
MAG 121/004 | Wed Mar 14 00:00:00 2018
code formatted:
while read line
do
case $line in
MAG*) echo -n $line "| " ;;
*) echo $line ;;
esac
done
This question already has answers here:
Bash script/command to print out date 5 min before/after
(4 answers)
Closed 5 years ago.
I want to add 10 seconds 10 times. But I don't know well how to add times to the value.
This is my code.
./time.sh
time=$(date)
counter=1
while [ $counter -le 10 ]
do
echo "$time"
time=$('$time + 10 seconds') //error occurred.
((counter++))
done
echo All done
Using GNU Date
Assuming GNU date, replace:
time=$('$time + 10 seconds')
with:
time=$(date -d "$time + 10 seconds")
Putting it all together, try:
$ cat a.sh
t=$(date)
counter=1
while [ "$counter" -le 10 ]
do
echo "$t"
t=$(date -d "$t + 10 seconds")
((counter++))
done
echo All done
(I renamed time to t because time is also a bash built-in command and it is best to avoid potential confusion.)
When run, the output looks like:
$ bash a.sh
Tue Jan 16 19:19:44 PST 2018
Tue Jan 16 19:19:54 PST 2018
Tue Jan 16 19:20:04 PST 2018
Tue Jan 16 19:20:14 PST 2018
Tue Jan 16 19:20:24 PST 2018
Tue Jan 16 19:20:34 PST 2018
Tue Jan 16 19:20:44 PST 2018
Tue Jan 16 19:20:54 PST 2018
Tue Jan 16 19:21:04 PST 2018
Tue Jan 16 19:21:14 PST 2018
All done
Using Bash (>4.2)
Recent versions of bash support date calculations without external utilities. Try:
$ cat b.sh
#!/bin/bash
printf -v t '%(%s)T' -1
counter=1
while [ "$counter" -le 10 ]
do
((t=t+10))
printf '%(%c)T\n' "$t"
((counter++))
done
echo All done
Here, t is time since epoch in seconds.
When run, the output looks like:
$ bash b.sh
Tue 16 Jan 2018 07:31:44 PM PST
Tue 16 Jan 2018 07:31:54 PM PST
Tue 16 Jan 2018 07:32:04 PM PST
Tue 16 Jan 2018 07:32:14 PM PST
Tue 16 Jan 2018 07:32:24 PM PST
Tue 16 Jan 2018 07:32:34 PM PST
Tue 16 Jan 2018 07:32:44 PM PST
Tue 16 Jan 2018 07:32:54 PM PST
Tue 16 Jan 2018 07:33:04 PM PST
Tue 16 Jan 2018 07:33:14 PM PST
All done
This is my first post here, thanks in advance for your prompt support.
I did some greps on a big file and came with file like the below. I want to make all lines after that starting with XX# in the same line.
So, I want the flowing:
XX#DEV1>
Feb 23 07:00:03
Feb 23 07:00:05
Sent : 4608
Received : 4227
Feb 23 07:00:07
Feb 23 07:00:09
XX#DEV2>
Feb 23 07:00:32
Feb 23 07:00:34
Sent : 4608
Received : 4232
Feb 23 07:00:36
Feb 23 07:00:38
XX#DEV1>
Feb 23 08:00:03
Feb 23 08:00:06
Sent : 4608
Received : 4265
Feb 23 08:00:07
Feb 23 08:00:09
XX#DEV2>
...
To become:
XX#DEV1> Feb 23 07:00:03 Feb 23 07:00:05 Sent : 4608 Received : 4227 Feb 23 07:00:07 Feb 23 07:00:09
XX#DEV2> Feb 23 07:00:32 Feb 23 07:00:34 Sent : 4608 Received : 4232 Feb 23 07:00:36 Feb 23 07:00:38
XX#DEV1> Feb 23 08:00:03 Feb 23 08:00:06 Sent : 4608 Received : 4265 Feb 23 08:00:07 Feb 23 08:00:09
XX#DEV2> ...
You can do that with awk pretty easily:
awk '/^XX/{if(length(out))print out;out=$0;next}{out=out" "$0}END{print out}' yourfile
That says.. if the line starts with "XX", print whatever we have accumluated in variable out, then save current line in variable out. If the line is anything else, append it to variable out after adding a space. At the end, print whatever we have accumulated in variable out.
Output:
XX#DEV1> Feb 23 07:00:03 Feb 23 07:00:05 Sent : 4608 Received : 4227 Feb 23 07:00:07 Feb 23 07:00:09
XX#DEV2> Feb 23 07:00:32 Feb 23 07:00:34 Sent : 4608 Received : 4232 Feb 23 07:00:36 Feb 23 07:00:38
XX#DEV1> Feb 23 08:00:03 Feb 23 08:00:06 Sent : 4608 Received : 4265 Feb 23 08:00:07 Feb 23 08:00:09
Or you can do it bash in much the same way as the awk in my other answer:
#!/bin/bash
while read line
do
if [[ $line == XX* ]] ;then
echo $OUT # Output whatever we have accumulated in $OUT
OUT=$line # Zero out $OUT and restart accumulating
else
OUT="$OUT $line" # Append this line to $OUT
fi
done < yourfile
echo $OUT # Output any trailing stuff at the end
sed -n '
# use -n to prevent sed from printing each line
# if the line read starts with XX, go (b) to :printit
/^XX/ b printit
# if it is the last line ($), also go to :printit
$ b printit
# for all other lines, append the read line to the hold space (H)
# and then go to the :end (so we read the next line)
H; b end
:printit {
# swap the hold and the pattern space
x
# replace all (s/../../g) single or multiple newlines (\n\n*)
# with a space
s/\n\n*/ /g
# print the pattern space
p
}
:end
' your_file