I have the following command line:
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 | awk 'NR==1{ips=$1} NR>1{ips=ips ", " $1} $2=="namenode"{nn=$1} END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}'
And that's producing a command line which I will use in my chef cookbook to start a MapR cluster:
/opt/mapr/server/configure.sh -C 10.32.237.251 -Z 10.32.237.251 -N mycluster --create-user -D /dev/xvdb
My first command only prints out that command. How can I modify that command, or add anything else, to automatically execute the command produced by the awk?
Pipe it to the shell:
egrep '^[0-9]' /etc/hosts | grep -v 127.0.0.1 \
| awk 'NR==1{ips=$1}
NR>1{ips=ips ", " $1}
$2=="namenode"{nn=$1}
END{ printf "/opt/mapr/server/configure.sh -C %s -Z %s -N mycluster --create-user -D /dev/xvdb\n", ips, nn}' \
| bash
Related
Context
Got a daft script that checks a process is running on a group of hosts, like a watchdog, as I say it's a daft script so bear in mind it isn't 'perfect' by scripting standards
Problem
I've ran bash -x and can see that the script finishes its first check without actually redirecting the output of the command to the file which is very frustrating, it means each host is actually being evaluated to the last hosts output
Code
#!/bin/bash
FILE='OUTPUT'
for host in $(cat /etc/hosts | grep webserver.[2][1-2][0-2][0-9] | awk {' print $2 ' })
do ssh -n -f $host -i <sshkey> 'ps ax | grep myprocess | wc -l' > $FILE 2> /dev/null
cat $FILE
if grep '1' $FILE ; then
echo "Process is NOT running on $host"
cat $FILE
else
cat $FILE
echo "ALL OK on $host"
fi
cat $FILE
done
Script traceback
++ cat /etc/hosts
++ awk '{ print $2 }'
++ grep 'webserver.[2][1-2][0-2][0-9]'
+ for host in '$(cat /etc/hosts | grep webserver.[2][1-2][0-2][0-9] | awk {'\'' print $2 '\''})'
+ ssh -n -f webserver.2100 -i <omitted> 'ps ax | grep myprocess | wc -l'
+ cat OUTPUT
+ grep 1 OUTPUT
+ cat OUTPUT
+ echo 'ALL OK on webserver.2100'
ALL OK on webserver.2100
+ cat OUTPUT
+ printf 'webserver.2100 checked \n'
webserver.2100 checked
+ for host in '$(cat /etc/hosts | grep webserver.[2][1-2][0-2][0-9] | awk {'\'' print $2 '\''})'
+ ssh -n -f webserver.2101 -i <omitted> 'ps ax | grep myprocess | wc -l'
+ cat OUTPUT
2
+ grep 1 OUTPUT
+ cat OUTPUT
2
+ echo 'ALL OK on webserver.2101'
ALL OK on webserver.2101
+ cat OUTPUT
2
+ printf 'webserver.2101 checked \n'
webserver.2101 checked
Issue
As you can see, it's registering nothing for the first host, then after it is done, it's piping the data into the file, then the second host is being evaluated for the previous hosts data...
I suspect its to do with redirection, but in my eyes this should work, it doesn't so it's frustrating.
I think you're assuming that ps ax | grep myprocess will always return at least one line (the grep process). I'm not sure that's true. I'd rewrite that like this:
awk '/webserver.[2][1-2][0-2][0-9]/ {print $2}' /etc/hosts | while IFS= read -r host; do
output=$( ssh -n -f "$host" -i "$sshkey" 'ps ax | grep "[m]yprocess"' )
if [[ -z "$output" ]]; then
echo "Process is NOT running on $host"
else
echo "ALL OK on $host"
fi
done
This trick ps ax | grep "[m]yprocess" effectively removes the grep process from the ps output:
the string "myprocess" matches the regular expression "[m]yprocess" (that's the running "myprocess" process), but
the string "[m]yprocess" does not match the regular expression "[m]yprocess" (that's the running "grep" process)
I'm trying to execute this command:
ps -eo pid,args --cols=10000 | awk '/\/opt\/logstash\/logstash-1.5.3\// && $1 != PROCINFO["pid"] { print $1 }'
whith ansible -m shell module (not working example):
ansible -m shell -a '"'ps -eo pid,args --cols=10000 | awk '/\/opt\/logstash\/logstash-1.5.3\// && $1 != PROCINFO[\'pid\'] { print $1 }' '"' all
One of the ways would be to put that into a file, but still it would be nice to run as a command - any ideas?
Bash escaping rules will do:
ansible localhost -m shell -a "ps -eo pid,args --cols=10000 | awk '/\\/opt\\/logstash\\/logstash-1.5.3\\// && \$1 != PROCINFO[\"pid\"] { print \$1 }'"
Mine alternative version that worked:
ansible -m command -a "ps a |grep -E '/opt/logstash/logstash-1.5.3/vendor/jruby' " all --sudo
Check if the process are running:
ansible -m shell -a "ps aux |grep -E '/opt/logstash/logstash-1.5.3/vendor/jruby'|grep -v -e grep |wc" all
A simple way (i.e. without having to rewrite the command much or introduce a bunch of escapes) is to use a temporary variable.
Your original command:
ps -eo pid,args --cols=10000 \
| awk '/\/opt\/logstash\/logstash-1.5.3\// && $1 != PROCINFO["pid"] { print $1 }'
Corresponding ansible call:
x='/\/opt\/logstash\/logstash-1.5.3\// && $1 != PROCINFO["pid"] { print $1 }'; \
ansible all -m shell -a "ps -eo pid,args --cols=10000 | awk '$x'"
Ok so I have the following script that updates Route43 DNS entries. Unfortunately there is a limit to the number of calls per second you can make so I need to make the final Xargs command sleep for about a second between each iteration.
I've tried a couple of things like ' {../cli53 blah; sleep 10; } ' and I cant seem to get it to work. Does anyone have any suggestions please:
#!/bin/bash
set root='dirname $0'
ec2-describe-instances -O ******* -W ******* --region eu-west-1 |
perl -ne '/^INSTANCE\s+(i-\S+).*?(\S+\.amazonaws\.com)/
and do { $dns = $2; print "$1 $dns\n" }; /^TAG.+\sName\s+(\S+)/
and print "$1 $dns\n"' |
perl -ane 'print "$F[0] CNAME $F[1] --replace\n"' |
grep -v 'i-' | xargs --verbose -n 4 /usr/local/bin/cli53 rrcreate -x 5 contoso.com
Edit: Thanks Etan for the Answer. Here is my solution for anyone else that needs it:
I had to include the -I %variable% switch into the xargs statement aswel to make sure that the feed in was passed as parameters to cli53 but it all looks to be working nicely now.
#!/bin/bash
set root='dirname $0'
ec2-describe-instances -O ******* -W ******* --region eu-west-1 |
perl -ne '/^INSTANCE\s+(i-\S+).*?(\S+\.amazonaws\.com)/
and do { $dns = $2; print "$1 $dns\n" }; /^TAG.+\sName\s+(\S+)/
and print "$1 $dns\n"' |
perl -ane 'print "$F[0] CNAME $F[1] --replace\n"' |
grep -v '^i-' |
xargs --verbose -n 4 -I myvar /bin/sh -c '{ /usr/local/bin/cli53 rrcreate -x 5 contoso.com 'myvar'; sleep 1; printf "\n\n"; }'
The simplest solution would be to simply put the cli53 and sleep calls in a script and use xargs to execute the script.
If you don't want to do that you should be able to do what you were trying to do with this:
... | xargs ... /bin/sh -c '{ /usr/local/bin/cli53 ... "$#"; sleep 10; }' -
i need to run hadoop command to list all live nodes, then based on the output i reformat it using awk command, and eventually output the result to a variable, awk use different delimiter each time i call it:
hadoop job -list-active-trackers | sort | awk -F. '{print $1}' | awk -F_ '{print $2}'
it outputs result like this:
hadoop-dn-11
hadoop-dn-12
...
then i put the whole command in variable to print out the result line by line:
var=$(sudo -H -u hadoop bash -c "hadoop job -list-active-trackers | sort | awk -F "." '{print $1}' | awk -F "_" '{print $2}'")
printf %s "$var" | while IFS= read -r line
do
echo "$line"
done
the awk -F didnt' work, it output result as:
tracker_hadoop-dn-1.xx.xsy.interanl:localhost/127.0.0.1:9990
tracker_hadoop-dn-1.xx.xsy.interanl:localhost/127.0.0.1:9390
why the awk with -F won't work correctly? and how i can fix it?
var=$(sudo -H -u hadoop bash -c "hadoop job -list-active-trackers | sort | awk -F "." '{print $1}' | awk -F "_" '{print $2}'")
Because you're enclosing the whole command in double quotes, your shell is expanding the variables $1 and $2 before launching sudo. This is what the sudo command looks like (I'm assuming $1 and $2 are empty)
sudo -H -u hadoop bash -c "hadoop job -list-active-trackers | sort | awk -F . '{print }' | awk -F _ '{print }'"
So, you see your awk commands are printing the whole line instead of just the first and 2nd fields respectively.
This is merely a quoting challenge
var=$(sudo -H -u hadoop bash -c 'hadoop job -list-active-trackers | sort | awk -F "." '\''{print $1}'\'' | awk -F "_" '\''{print $2}'\')
A bash single quoted string cannot contain single quotes, so that's why you see ...'\''... -- to close the string, concatenate a literal single quote, then re-open the string.
Another way is to escape the vars and inner double quotes:
var=$(sudo -H -u hadoop bash -c "hadoop job -list-active-trackers | sort | awk -F \".\" '{print \$1}' | awk -F \"_\" '{print \$2}'")
How can i take the output of this command...
ps -ef | grep ^apache | grep /sbin/httpd | awk '{print $2}'
16779
16783
16784
16785
16786
16787
16788
16789
16790
16794
16795
16796
16797
16799
16800
16801
16802
16803
16804
16805
...so a single column of numbers... and transform those numbers into a single line of numbers separated by a " -p "... This would be used for the following...
lsof -p 16779 -p 16783 -p 16784 ...
Pipe into
sed 's/^/-p /' | tr '\n' ' '
If you have it available, pidof would be more convenient:
lsof $(pidof apache | sed 's/^\| / -p /g')
You could pipe into awk:
awk 'BEGIN { printf "lsof" } { printf " -p %s", $1 } END { printf "\n" }'
Result:
lsof -p 16779 -p 16783 -p 16784 -p 16785 -p 16786 -p 16787 -p 16788 -p 16789 -p 16790 -p 16794 -p 16795 -p 16796 -p 16797 -p 16799 -p 16800 -p 16801 -p 16802 -p 16803 -p 16804 -p 16805
tmp="lsof "
for i in `ps -ef | awk '/^apache/ && /httpd/ {print $2}'`;
do
tmp=${tmp}" -p "${i}" ";
done
echo $tmp
Should do the trick
In a command substitution, the newlines from the pipeline will be converted to spaces.
pids=$( ps -ef | awk '/^apache/ && /\/sbin\/httpd/ {print $2}' ) )
Then a call to printf can be used to format the options for lsof. The format string is repeated as necessary for each argument contained in pids.
lsof $( printf "-p %s " $pids )
Add the following code to your one liner:
awk '{print $0 " -p "}' | tr '\n' ' ' | awk -F " " '{print "lsof -p " $0}'
Final code :
ps -ef | grep ^apache | grep /sbin/httpd | awk '{print $2}' | awk '{print $0 " -p "}' | tr '\n' ' ' | awk -F " " '{print "lsof -p " $0}'