I would like to get the total memory of a host by using an Expect script. Thanks to the answers I received in Get Total Memory of a host with LINUX/EXPECT I am now closer to the final solution. (Thanks to glenn jackman and Sorpigal).
After connecting to the host, I execute the following line:
send "cat /proc/meminfo | grep MemTotal | awk '{print \$2}'\r"
It returns the value I want, but when I get the value contained in $expect_out(buffer), it contains much more info than I need, including the send sentences and the previous and posterior command prompt flags. I exactly get the following, using exp_internal 1:
expect: set expect_out(buffer) "17# cat /proc/meminfo | grep MemTotal | awk '{print $2}'\r\n34150400\r\nCBA"
I want to extract just the number near the end, 34150400. Any hint or idea?
regexp "\n(\[0-9]+)" $expect_out(buffer) - num
puts $num ;# => 34150400
By the way, you can do this without sending a shell command:
set fid [open /proc/meminfo]
while {[gets $fid line] != -1} {
if {[regexp {^MemTotal: *(\d+)} $line - memtotal]} {
break
}
}
close $fid
puts $memtotal
Related
I have file abc.sh which contains below data -
a_a_1 was unsuccessful
a_a_5 was completed
a_a_2 was unsuccessful
a_a_4 was unsuccessful
a_a_9 was unsuccessful
now, I have a variable abc which contains value 2,1,9 ..i.e abc=2,1,9
now want to print only those lines from file which matches value 2,1,9 and above string.
output should be like-
a_a_2 was unsuccessful
a_a_1 was unsuccessful
a_a_9 was unsuccessful
How to achieve above output?
Since this is tagged tcl...
#!/usr/bin/env tclsh
proc main {abc} {
set abc [string map {, |} $abc]
set re [string cat {^a_a_(?:} $abc {)\M}]
while {[gets stdin line] >= 0} {
if {[regexp $re $line]} {
puts $line
}
}
}
main [lindex $argv 0]
Example usage:
$ ./findit.tcl 2,1,9 < abc.sh
Basically, it converts the CSV 2,1,9 into pipe delimited 2|1|9 and uses that as part of a bigger regular expression, and prints lines read from standard input that match it.
Since you also seem to be interested in a bash solution (you already got one for Tcl):
grep -E "_(${abc//,/|}) " abc.sh
The idea here is to translate the 2,1,9 into the regexp pattern 2|1|9. An alternative, similar in spirit, would be
grep "_[${abc//,/}] " abc.sh
which produces the pattern [219].
Your variable need to be:
abc="2\|1\|9" for grep $abc abc.sh
a=2 b=1 c=9 for grep "[$a]\|[$b]\|[$c]" abc.sh
Well as your question, if you want for abc=2,1,9 , you can just change the , as \| using sed when executing grep like this:
grep $(echo $abc | sed "s/,/\\\|/g") abc.sh
*ps: English is not my primary language so please excuse any grammar mistakes :)
2001-06-30T11:33:33,543 DEBUG (Bss-Thread-948:[]) SUNCA#44#77#CALMED#OK#58#NARDE#4356#68654768961#BHR#TST#DEV
2001-06-30T11:33:33,543 DEBUG (Bss-Thread-948:[]) SUNCA#44#77#CALMED#OK#58#NARDE#89034#1234567#BHR#TST#DEV
2001-06-30T11:33:33,543 DEBUG (Bss-Thread-948:[]) SUNCA#44#77#OK#58#BHREDD#234586#4254567#BHR#TST#DEV
2001-06-30T11:33:33,543 DEBUG (Bss-Thread-948:[]) SUNCA#44#77#OK#58#NARDE#89034#1034567#BHR#TST#DEV
I have log file mentioned above. I would like to print lines only if value between patterns # and #BHR is greater than 1100000.
I can see in my log file lines with values 68654768961, 1234567, 4254567, 1034567. As per the requirement the output should conatin only first 3 lines.
I am looking for regex to get desired output.
One questions, this #58#BHR should be ignore in third line ? If yes, I will get value between patterns # and #BHR#.
Normally, it should be solved this question by writing scripting according the business logical. But you could try this one line command by awk.
awk '{if (0 == system("[ $(echo \"" $0 "\"" " | grep -oP \"" "(?<=#)\\d+(?=#BHR#)\" || echo 0) -gt 1100000 ]")) {print $0}}' log_file
Mainly, it use system() to scratch the value by grep:
# if can't get the pattern value by grep, the value will assign 0
echo $one_line | grep -oP "(?<=#)\d+(?=#BHR#)" || echo 0`
and compare the value to 1100000 by [ "$value" -gt 1100000 ] in awk.
FYI, so if the value greater than 1100000 it will return 0.
system(cmd): executes cmd and returns its exit status
I have wrote a script below to parse a text file that effectively removes line returns. It will take input that looks like this:
TCP 0.0.0.0:135 SVR LISTENING 776
RpcSs
And return this to a new text document
TCP 0.0.0.0:135 SVR LISTENING 776 RpcSs
Some entries span more than two lines so I was not able to write a script that removes the line return from every other line so I came up with this approach below. It worked fine for small collects but a 7MB collect resulted in my computer running out of memory and it took quite a bit of time do this before it failed. I'm curious why it ran out of memory as well as hoping someone could educate me on a better way to do this.
#!/bin/bash
#
# VARS
writeOuput=""
#
while read line
do
curLine=$line #grab current line from document
varWord=$(echo $curLine | awk '{print $1}') #grab first word from each line
if [ "$varWord" == "TCP" ] || [ "$varWord" == "UDP" ]; then
#echo "$curLine" >> results.txt
unset writeOutput
writeOutput=$curLine
elif [ "$varWord" == "Active" ]; then #new session
printf "\n" >> results1.txt
printf "New Session" >> results1.txt
printf "\n" >> results1.txt
else
writeOutput+=" $curLine"
#echo "$writeOutput\n"
printf "$writeOutput\n" >> results1.txt
#sed -e '"$index"s/$/"$curLine"'
fi
done < $1
Consider replacing the line with the awk call with this line:
varWord=${curLine%% *} #grab first word from each line
This saves the fork that happens in each iteration by using Bash-internal functionality only and should make your program run several times faster. See also that other guy's comment linking to this answer for an explanation.
As others have noted, the main bottleneck in your script is probably the forking where you pass each line through its own awk instance.
I have created an awk script which I hope does the same as your bash script, and I suspect it should run faster. Initially I just thought about replacing newlines with spaces, and manually adding newlines in front of every TCP or UDP, like this:
awk '
BEGIN {ORS=" "};
$1~/(TCP|UDP)/ {printf("\n")};
{print};
END {printf("\n")}
' <file>
But your script removes the 'Active' lines from the output, and adds three new lines before the line. You could, of course, pipe this through a second `awk command:
awk '/Active/ {gsub(/Active /, ""); print("\nNew Session\n")}; {print}'
But this awk script is a bit closer to what you did with bash, but it should still be considerably faster:
$ cat join.awk
$1~/Active/ {print("\nNew Session\n"); next}
$1~/(TCP|UDP)/ {if (output) print output; output = ""}
{if (output) output = output " " $0; else output = $0}
END {print output}
$ awk -f join.awk <file>
First, it checks whether the line begins with the word "Active", if it does, it prints the three lines, and goes on to the next input line.
Otherwise it checks for the presence of TCP or UDP as the first word. If it finds them, it prints what has accumulated in writeOutput (provided there is something in the variable), and clears it.
It then adds whatever it finds in the line to writeOutput
At the end, it prints what has accumulated since the last TCP or UDP.
Normally I use something like this for processes I run on my servers
./runEvilProcess.sh >> ./evilProcess.log
However I'm currently using Doxygen and it produces lots of duplicate output
Example output:
QGDict::hashAsciiKey: Invalid null key
QGDict::hashAsciiKey: Invalid null key
QGDict::hashAsciiKey: Invalid null key
So you end up with a very messy log
Is there a way I can only add the line to the log file if the line wasn't the last one added.
A poor example (but not sure how to do in bash)
$previousLine = ""
$outputLine = getNextLine()
if($previousLine != $outputLine) {
$outputLine >> logfile.log
$previousLine = $outputLine
}
If the process returns duplicate lines in a row, pipe the output of your process through uniq:
$ ./t.sh
one
one
two
two
two
one
one
$ ./t.sh | uniq
one
two
one
If the logs are sent to the standard error stream, you'll need to redirect that too:
$ ./yourprog 2>&1 | uniq >> logfile
(This won't help if the duplicates come from multiple runs of the program - but then you can pipe your log file through uniq when reviewing it.)
Create a filter script (filter.sh):
while read line; do
if [ "$last" != "$line" ]; then
echo $line
last=$line
fi
done
and use it:
./runEvilProcess.sh | sh filter.sh >> evillog
1. File
A file /etc/ssh/ipblock contains lines that look like this:
2012-01-01 12:00 192.0.2.201
2012-01-01 14:15 198.51.100.123
2012-02-15 09:45 192.0.2.15
2012-03-12 21:45 192.0.2.14
2012-04-25 00:15 203.0.113.243
2. Command
The output of the command iptables -nL somechain looks like this:
Chain somechain (2 references)
target prot opt source destination
DROP all -- 172.18.1.4 anywhere
DROP all -- 198.51.100.123 anywhere
DROP all -- 172.20.4.16 anywhere
DROP all -- 192.0.2.125 anywhere
DROP all -- 172.21.1.2 anywhere
3. The task at hand
First I would like to get a list A of IP addresses that are existent in the iptables chain (field 4) but not in the file.
Then I would like to get a list B of IP addresses that are existent in the file but not in the iptables chain.
IP addresses in list A should then be appended to the file in the same style (date, time, IP)
IP addresses in list B should then be added to the iptables chain with
iptables -A somechain -d IP -j DROP
4. Background
I was hoping to expand my awk-fu so I have been trying to get this to work with an awk script that can be executed without arguments. But I failed.
I know I can get the output from commands with the getline command so I was able to get the time and date that way. And I also know that one can read a file using getline foo < file. But I have only had many failed attempts to combine this all into a working awk script.
I realise that I could get this to work with an other programming language or a shell script. But can this be done with an awk script that can be ran without arguments?
I think this is almost exactly what you were looking for. Does the job, all in one file, code I guess is pretty much self-explanatory...
Easily adaptable, extendable...
USAGE:
./foo.awk CHAIN ip.file
foo.awk:
#!/usr/bin/awk -f
BEGIN {
CHAIN= ARGV[1]
IPBLOCKFILE = ARGV[2]
while((getline < IPBLOCKFILE) > 0) {
IPBLOCK[$3] = 1
}
command = "iptables -nL " CHAIN
command |getline
command |getline
while((command |getline) > 0) {
IPTABLES[$4] = 1
}
close(command)
print "not in IPBLOCK (will be appended):"
command = "date +'%Y-%m-%d %H:%M'"
command |getline DATE
close(command)
for(ip in IPTABLES) {
if(!IPBLOCK[ip]) {
print ip
print DATE,ip >> IPBLOCKFILE
}
}
print "not in IPTABLES (will be appended):"
# command = "echo iptables -A " CHAIN " -s " //use for testing
command = "iptables -A " CHAIN " -s "
for(ip in IPBLOCK) {
if(!IPTABLES[ip]) {
print ip
system(command ip " -j DROP")
}
}
exit
}
Doing 1&3:
comm -13 <(awk '{print $3}' /etc/ssh/ipblock | sort) <(iptables -nL somechain | awk '/\./{print $4}' | sort) | xargs -n 1 echo `date '+%y-%m-%d %H:%M'` >> /etc/ipblock
Doing 2&4:
comm -13 <(awk '{print $3}' /etc/ssh/ipblock | sort) <(iptables -nL somechain | awk '/\./{print $4}' | sort) | xargs -n 1 iptables -A somechain -d IP -j DROP
The command is constructed of the following building blocks:
Bash process substitution feature: it is somewhat similar to pipe features, but is often used when a program requires two or more input files in its arguments/options. Bash creates fifo file, which basically "contains" the output of a given command. In our case the output will be ip adresses.
Then output of awk scripts is passed to comm program, and both awk scripts are pretty simple: they just print ip address. In first case all ips are contained in third column(hence $3), and in the second case all ips are contained in the fourth column, but it is neccessary to get rid of column header("destination" string), so simple regex is used /\./: it filters out all string that doesn't contain a dot.
comm requires both inputs to be sorted, thus output of awk is sorted using sort
Now comm program receives both lists of ip addresses. When no options are given, it prints three columns: lines unique to FILE1, lines unique to FILE2, lines in both files. By passing -23 to it we get only lines unique to FILE1. Similarly, passing -13 makes it output lines unique to FILE2.
xargs is basically a "foreach" loop in bash, it executes a given command per each input line(thanks to -n 1). The second is pretty obvious(it is the desired iptables invocation). The second one isn't complicated too: it just makes date to output current time in proper format.