I am trying to make a single line ssh call from a ruby script. My script takes a hostname, and then sets out to return the hostname's machine info.
return_value = %x{ ssh #{hostname} "#{number_of_users}; #{number_of_processes};
#{number_of_processes_running}; #{number_of_processes_sleeping}; "}
Where the variables are formatted like this.
number_of_users = %Q(users | wc -w | cat | awk '{print "Number of Users: "\$1}')
number_of_processes = %Q(ps -el | awk '{print $2}' | wc -l | awk '{print "Number of Processes: "$1}')
I have tried both %q, %Q, and just plain "" and I cannot get the awk to print anything before the output. I either get this error (if I include the colon)
awk: line 1: syntax error at or near :
or if I don't include the slash in front of $1 I just get empty output for that line. Is there any solution for this? I thought it might be because I was using %q, but it even happens with just double quotes.
Use backticks to capture the output of the command and return the output as a string:
number_of_users = `users | wc -w | cat | awk '{print "Number of Users:", $1}'`
puts number_of_users
Results on my system:
48
But you can improve your pipeline:
users | awk '{ print "Number of Users:", NF }'
ps -e | awk 'END { print "Number of Processes:", NR }'
So the solution to this problem is:
%q(users | wc -w | awk '{print \"Number of Users: \"\$1}')
Where you have to use %q, not %, not %Q, and not ""
You must backslash double quotes and the dollar sign in front of any awk variables
If somebody could improve upon this answer by explaining why, that would be most appreciated
Though as Steve pointed out I could have improved my code using users | awk '{ print \"Number of Users:\", NF }'
In which case there is no need to backslash the NF.
Related
I tried to parse a text file having delimiter as "|^|" using awk command. But awk command is not working as expected.
Below is the example:
Command-1:
echo "28851|^|178838|^||^|" | awk -F '|^|' '{ print $1}'
Output:
28851|^|178838|^||^|
Expected output:
28851.
Command-2:
echo "28851|^|178838|^||^|" | awk -F '|^|' '{ print $2}'
Output:
BLANK or NULL
Expected output:
178838
Please provide some inputs on how to parse the text file in Unix.
Awk treats |^| as a complex regex pattern. But since | and ^ are regex metacharacters - they should be escaped with \ (\\|\\^\\|) or put into a character classes [|][^][|].
echo "28851|^|178838|^||^|" | awk -F'\\|\\^\\|' '{ print $2 }'
178838
echo '28851|^|178838|^||^|' | awk -F'\\|\\^\\|' '{print $2}'
You must escape special characters.
The string format is
Executed: variable_name
What is the simplest way to get the *variable_name* sub-string?
foo="Executed: variable_name"
echo ${foo##* } # Strip everything up to and including the rightmost space.
or
set -- $foo
echo $2
Unlike other solutions using awk, sed, and whatnot, these don't fork other programs and save thousands of CPU cycles since they execute completely in your shell. They are also more portable (unlike ${i/* /} which is a bashism).
With sed:
echo "Executed: variable_name" | sed 's/[^:]*: //'
Using awk:
echo 'Executed: variable_name' | awk -F' *: *' '{print $2}'
variable_name
If you have the string in a variable:
$ i="Executed: variable_name"
$ echo ${i/* /}
variable_name
If you have the string as output of a command
$ cmd
Executed: variable_name
$ cmd | awk '{print $NF}'
variable_name
Note that 'NF' means "number of fields", so $NF is always the last field on the line. Fields are assumed to be separated by spaces (unless -F is specified)
If your variable_name could have spaces in it, the
-F' *: *'
mentioned previously ensures that only the ": " is used as a field separator. However, this will preserve spaces at the end of the line if there are any.
If the line is mixed in with other output, you might need to filter.
Either grep..
$ cmd | grep '^Executed: ' | awk '{print $NF}'
or more clever awk..
$ cmd | awk '/^Executed: /{print $NF}'
I am trying to run a shell command from within awk for each line of a file, and the shell command needs one input argument. I tried to use system(), but it didn't recognize the input argument.
Each line of this file is an address of a file, and I want to run a command to process that file. So, for a simple example I want to use 'wc' command for each line and pass $1to wc.
awk '{system("wc $1")}' myfile
you are close. you have to concatenate the command line with awk variables:
awk '{system("wc "$1)}' myfile
You cannot grab the output of an awk system() call, you can only get the exit status. Use the getline/pipe or getline/variable/pipe constructs
awk '{
cmd = "your_command " $1
while (cmd | getline line) {
do_something_with(line)
}
close(cmd)
}' file
FYI here's how to use awk to process files whose names are stored in a file (providing wc-like functionality in this example):
gawk '
NR==FNR { ARGV[ARGC++]=$0; next }
{ nW+=NF; nC+=(length($0) + 1) }
ENDFILE { print FILENAME, FNR, nW, nC; nW=nC=0 }
' file
The above uses GNU awk for ENDFILE. With other awks just store the values in an array and print in a loop in the END section.
I would suggest another solution:
awk '{print $1}' myfile | xargs wc
the difference is that it executes wc once with multiple arguments. It often works (for example, with kill command)
Or use the pipe | as in bash then retrive the output in a variable with awk's getline, like this
zcat /var/log/fail2ban.log* | gawk '/.*Ban.*/ {print $7};' | sort | uniq -c | sort | gawk '{ "geoiplookup " $2 "| cut -f2 -d: " | getline geoip; print $2 "\t\t" $1 " " geoip}'
That line will print all the banned IPs from your server along with their origin (country) using the geoip-bin package.
The last part of that one-liner is the one that affects us :
gawk '{ "geoiplookup " $2 "| cut -f2 -d: " | getline geoip; print $2 "\t\t" $1 " " geoip}'
It simply says : run the command "geoiplookup 182.193.192.4 | -f2 -d:" ($2 gets substituted as you may guess) and put the result of that command in geoip (the | getline geoip bit). Next, print something something and anything inside the geoip variable.
The complete example and the results can be found here, an article I wrote.
I am working on the following bash script:
# contents of dbfake file
1 100% file 1
2 99% file name 2
3 100% file name 3
#!/bin/bash
# cat out data
cat dbfake |
# select lines containing 100%
grep 100% |
# print the first and third columns
awk '{print $1, $3}' |
# echo out id and file name and log
xargs -rI % sh -c '{ echo %; echo "%" >> "fake.log"; }'
exit 0
This script works ok, but how do I print everything in column $3 and then all columns after?
You can use cut instead of awk in this case:
cut -f1,3- -d ' '
awk '{ $2 = ""; print }' # remove col 2
If you don't mind a little whitespace:
awk '{ $2="" }1'
But UUOC and grep:
< dbfake awk '/100%/ { $2="" }1' | ...
If you'd like to trim that whitespace:
< dbfake awk '/100%/ { $2=""; sub(FS "+", FS) }1' | ...
For fun, here's another way using GNU sed:
< dbfake sed -r '/100%/s/^(\S+)\s+\S+(.*)/\1\2/' | ...
All you need is:
awk 'sub(/.*100% /,"")' dbfake | tee "fake.log"
Others responded in various ways, but I want to point that using xargs to multiplex output is rather bad idea.
Instead, why don't you:
awk '$2=="100%" { sub("100%[[:space:]]*",""); print; print >>"fake.log"}' dbfake
That's all. You don't need grep, you don't need multiple pipes, and definitely you don't need to fork shell for every line you're outputting.
You could do awk ...; print}' | tee fake.log, but there is not much point in forking tee, if awk can handle it as well.
Is there an easy way to print the IP_Address:port# ? Because I soon as it gets to the SED command, the port :# is stripped
input file example
Apr 6 14:20:41 TCP 178.255.83.1:80 in
preferred output like this
Apr 6 14:20:41 TCP 178.255.83.1:80 in United Kingdom
egrep -w 'TCP|UDP' $Denied_IPs |
sed 's/:[^:]* in/ in/; s/:[^:]* out/ out/' |
awk '{cmd="echo "$5" | code | fgrep 'Country:' | cut -c 16-43";
cmd | getline rslt;
close(cmd);
print $1" "$2" "$3" "$4" "$5" "$6" "rslt}' >> "$IP2COUNTRY"
The sed command is stripping the port explicitly. Given that that is all the sed command is doing, simply remove it from the expression.
That's a rather unoptimal implementation, by the way. Especially after we remove the sed, the egrep can be folded into the awk:
awk '/ (TCP|UDP) / {
split($5, addr, /:/);
cmd = "echo " addr[1] " | code | fgrep Country: | cut -c 16-43";
cmd | getline rslt;
close(cmd);
print $1, $2, $3, $4, $5, $6, rslt
}' < "$Denied_IPs" >> "$IP2COUNTRY"
and I can't help but think that the invocation of code within awk can be optimized a bit.
(I also removed the single quotes around 'Country:', which were doing nothing useful — and if they had been needed, they would in fact have broken the script because the whole thing is already wrapped in single quotes.)