Ok so I have the following script that updates Route43 DNS entries. Unfortunately there is a limit to the number of calls per second you can make so I need to make the final Xargs command sleep for about a second between each iteration.
I've tried a couple of things like ' {../cli53 blah; sleep 10; } ' and I cant seem to get it to work. Does anyone have any suggestions please:
#!/bin/bash
set root='dirname $0'
ec2-describe-instances -O ******* -W ******* --region eu-west-1 |
perl -ne '/^INSTANCE\s+(i-\S+).*?(\S+\.amazonaws\.com)/
and do { $dns = $2; print "$1 $dns\n" }; /^TAG.+\sName\s+(\S+)/
and print "$1 $dns\n"' |
perl -ane 'print "$F[0] CNAME $F[1] --replace\n"' |
grep -v 'i-' | xargs --verbose -n 4 /usr/local/bin/cli53 rrcreate -x 5 contoso.com
Edit: Thanks Etan for the Answer. Here is my solution for anyone else that needs it:
I had to include the -I %variable% switch into the xargs statement aswel to make sure that the feed in was passed as parameters to cli53 but it all looks to be working nicely now.
#!/bin/bash
set root='dirname $0'
ec2-describe-instances -O ******* -W ******* --region eu-west-1 |
perl -ne '/^INSTANCE\s+(i-\S+).*?(\S+\.amazonaws\.com)/
and do { $dns = $2; print "$1 $dns\n" }; /^TAG.+\sName\s+(\S+)/
and print "$1 $dns\n"' |
perl -ane 'print "$F[0] CNAME $F[1] --replace\n"' |
grep -v '^i-' |
xargs --verbose -n 4 -I myvar /bin/sh -c '{ /usr/local/bin/cli53 rrcreate -x 5 contoso.com 'myvar'; sleep 1; printf "\n\n"; }'
The simplest solution would be to simply put the cli53 and sleep calls in a script and use xargs to execute the script.
If you don't want to do that you should be able to do what you were trying to do with this:
... | xargs ... /bin/sh -c '{ /usr/local/bin/cli53 ... "$#"; sleep 10; }' -
Related
Context
Got a daft script that checks a process is running on a group of hosts, like a watchdog, as I say it's a daft script so bear in mind it isn't 'perfect' by scripting standards
Problem
I've ran bash -x and can see that the script finishes its first check without actually redirecting the output of the command to the file which is very frustrating, it means each host is actually being evaluated to the last hosts output
Code
#!/bin/bash
FILE='OUTPUT'
for host in $(cat /etc/hosts | grep webserver.[2][1-2][0-2][0-9] | awk {' print $2 ' })
do ssh -n -f $host -i <sshkey> 'ps ax | grep myprocess | wc -l' > $FILE 2> /dev/null
cat $FILE
if grep '1' $FILE ; then
echo "Process is NOT running on $host"
cat $FILE
else
cat $FILE
echo "ALL OK on $host"
fi
cat $FILE
done
Script traceback
++ cat /etc/hosts
++ awk '{ print $2 }'
++ grep 'webserver.[2][1-2][0-2][0-9]'
+ for host in '$(cat /etc/hosts | grep webserver.[2][1-2][0-2][0-9] | awk {'\'' print $2 '\''})'
+ ssh -n -f webserver.2100 -i <omitted> 'ps ax | grep myprocess | wc -l'
+ cat OUTPUT
+ grep 1 OUTPUT
+ cat OUTPUT
+ echo 'ALL OK on webserver.2100'
ALL OK on webserver.2100
+ cat OUTPUT
+ printf 'webserver.2100 checked \n'
webserver.2100 checked
+ for host in '$(cat /etc/hosts | grep webserver.[2][1-2][0-2][0-9] | awk {'\'' print $2 '\''})'
+ ssh -n -f webserver.2101 -i <omitted> 'ps ax | grep myprocess | wc -l'
+ cat OUTPUT
2
+ grep 1 OUTPUT
+ cat OUTPUT
2
+ echo 'ALL OK on webserver.2101'
ALL OK on webserver.2101
+ cat OUTPUT
2
+ printf 'webserver.2101 checked \n'
webserver.2101 checked
Issue
As you can see, it's registering nothing for the first host, then after it is done, it's piping the data into the file, then the second host is being evaluated for the previous hosts data...
I suspect its to do with redirection, but in my eyes this should work, it doesn't so it's frustrating.
I think you're assuming that ps ax | grep myprocess will always return at least one line (the grep process). I'm not sure that's true. I'd rewrite that like this:
awk '/webserver.[2][1-2][0-2][0-9]/ {print $2}' /etc/hosts | while IFS= read -r host; do
output=$( ssh -n -f "$host" -i "$sshkey" 'ps ax | grep "[m]yprocess"' )
if [[ -z "$output" ]]; then
echo "Process is NOT running on $host"
else
echo "ALL OK on $host"
fi
done
This trick ps ax | grep "[m]yprocess" effectively removes the grep process from the ps output:
the string "myprocess" matches the regular expression "[m]yprocess" (that's the running "myprocess" process), but
the string "[m]yprocess" does not match the regular expression "[m]yprocess" (that's the running "grep" process)
This code ouputs a http status of 000 - which seems to indicate something didn't connect properly but when I do this curl outside of the bash script it works fine and produces a 200 so something with this code is off... any guidance?
#!/bin/bash
URLs=$(< test.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do curl -L -s -w "%{http_code} %{url_effective}\\n" $line
done
http://beerpla.net/2010/06/10/how-to-display-just-the-http-response-code-in-cli-curl/
your script works on my vt.
I added in a couple of debugging lines, this may help you to see where any metacharacters are getting in, as I would have to agree with the posted coments.
I've output lines in the for to a file which is then printed out with od.
I have amended the curl line to grab the last line, just to get the response code.
#!/bin/bash
echo -n > $HOME/Desktop/urltstfile # truncate urltstfile
URLs=$(cat testurl.txt | grep Url | awk -F\ ' { print $2 } ')
# printf "Preparing to check $URLs \n"
for line in $URLs
do echo $line >> $HOME/Desktop/urltstfile;
echo line:$line:
curl -IL -s -w "%{http_code}\n" $line | tail -1
done
od -c $HOME/Desktop/urltstfile
#do curl -L -s -w "%{http_code} %{url_effective}\\n" "$line\n"
I have the following script which sends the results of an iwlist scan via OSC:
#!/bin/bash
NUM_BANKS=20
while [[ "$input" != "\e" ]] ; do
networks=$(iwlist wlan0 scanning | awk 'BEGIN{ FS="[:=]"; OFS = " " }
/ESSID/{
#gsub(/ /,"\\ ",$2)
#gsub(/\"/,"",$2)
essid[c++]=$2
}
/Address/{
gsub(/.*Address: /,"")
address[a++]=$0
}
/Encryption key/{ encryption[d++]=$2 }
/Quality/{
gsub(/ dBm /,"")
signal[b++]=$3
}
END {
for( c in essid ) { print "/wlan_scan ",essid[c],signal[c],encryption[c] }
}'
)
read -t 0.1 input
echo "$networks" | while read network; do
set $network
hash=` echo "$2" | md5sum | awk '{ print $1 }'| tr '[:lower:]' '[:upper:]'`
bank=`echo "ibase=16;obase=A; $hash%$NUM_BANKS " | bc`
echo "$1$bank $2 $3 $4"
echo "$1$bank $2 $3 $4" | sendOSC -h localhost 9997
done
#echo "$networks" | sendOSC -h localhost 9997
done
An example of the output from this is '/wlan_scan13 BTHomehub757 -85 On', which is then sent via the sendOSC program.
I basically need to replace the iwlist scan data with the results of this tshark scan:
sudo tshark -I -i en1 -T fields -e wlan.sa_resolved -e wlan_mgt.ssid -e radiotap.dbm_antsignal type mgt subtype probe
which similarly outputs two strings and an int, outputting a result like:
'Hewlett-_91:fa:xx EE-BrightBox-mjmxxx -78'.
So eventually I want the script to give me an output in this instance of
'/wlan13 Hewlett-_91:fa:xx EE-BrightBox-mjmxxx -78'.
Both scans constantly generate results in this format at about the same rate, updating as new wifi routers are detected, and these are sent out as soon as they arrive over the sendOSC program.
This is probably a pretty simple edit for an experienced coder, but I've been trying to work this out for days and I figured I should ask for help!
If someone could clarify what needs to stay and what needs to go here I'd really appreciate it.
Many thanks.
Do you really want to replace commands? The sane approach would seem to be to add an option to the script to specify which piece of code to run, and include them both.
# TODO: replace with proper option parsing
case $1 in
--tshark) command=tshark_networks; shift;;
*) command=iwlist_networks;;
esac
tshark_networks () {
sudo tshark -I -i en1 -T fields \
-e wlan.sa_resolved \
-e wlan_mgt.ssid \
-e radiotap.dbm_antsignal type mgt subtype probe
}
iwlist_networks () {
iwlist wlan0 scanning | awk .... long Awk script here ....
}
while [[ "$input" != "\e" ]] ; do
networks=$($command)
read -t 0.1 input
echo "$networks" | while read network; do
: the rest as before, except fix your indentation
This also has the nice side effect that the hideous iwlist command is encapsulated in its own function, outside of the main loop.
... Well, in fact, I might refactor the main loop to
while true; do
$command |
while read a b c d; do
hash=$(echo "$b" | md5sum | awk '{ print toupper($1) }')
bank=$(echo "ibase=16;obase=A; $hash%$NUM_BANKS " | bc)
echo "$a$bank $b $c $d"
echo "$a$bank $b $c $d" | sendOSC -h localhost 9997
done
read -t 0.1 input
case $input in '\e') break;; esac
done
I have a line working from this thread that tails a file until a matching pattern is found. It works well, but I can't find a way to suppress the output that occurs afterwards.
The line is:
sh -c 'tail -n +0 -f $logfile | { sed "/EOF/ q" && kill $$ ;}'
piping to /dev/null doesn't work as I don't get any output at all from the tail command that way. Also, I'm on OSX and various other sed and awk suggestions don't work due to the syntax.
It always finishes with the below, instead of nothing:
sh: line 10: 14285 Terminated: 15 sh -c 'tail -n +0 -f $logfile | { sed "/EOF/ q" && kill $$ ;}'
I'd also like not to display the matched text (EOF in the above example).
Any suggestions welcomed.
for a file (log for example)
sed -u "/pattern/ q" YourFile
for a pipe
ls -l | sed -u "/pattern/ q"
the -u of sed tell it to work as a stream input
It's the shell's job monitoring popping the message.
nomonitor() {
set +m
"$#"
set -m
}
nomonitor sh -c 'tail -n +0 -f $logfile | { sed "/EOF/ q" && kill $$; }'
You can actually discard the stderr like this:
sh -c 'tail -n +0 -f $logfile | { sed "/EOF/q" && p=$$ && kill $((p+1)) ; }'
This is my bash script:
#!/usr/local/bin/bash -x
touch /usr/local/p
touch /usr/local/rec
DATA_FULL=`date +%Y.%m.%d.%H`
CHECK=`netstat -an | grep ESTAB | egrep '(13001|13002|13003|13004|13061|13099|16001|16002|16003|16004|16061|16099|18001|18002|18003|18004|18061|18099|20001|20002|20003|20004|20061|20099|13000|16000|18000|20000)' | awk '{ print $5 }' | sort -u | wc -l`
netstat -an | grep ESTAB | egrep '(13001|13002|13003|13004|13061|13099|16001|16002|16003|16004|16061|16099|18001|18002|18003|18004|18061|18099|20001|20002|20003|20004|20061|20099|13000|16000|18000|20000)' | awk '{ print $5 }' | sort -u | wc -l > /usr/local/www/p
STAT=`cat /usr/local/www/rec`
if [ "$CHECK" -gt "$STAT" ]; then
echo $CHECK"\n"$DATA_FULL > /usr/local/p
fi
Ofcourse I've runned chmod +x script.sh and then sh script.sh, then I receive the following message: [: : bad number.
Why does it happends?
Run your script using
sh -x script.sh
It'll print every line it executes and the variable output.
Run the netstat command and stat command outside and check.
If these are integer for sure, use this syntax,
if [ "0$(echo $CHECK|tr -d ' ')" -gt "0$(echo $STAT|tr -d ' ')" ];
A simple hack. Only works if $STAT is always either empty or positive number.
Are you sure that both STAT and CHECK are numbers that can be compared with -gt?
probably your /usr/local/www/rec is empty. Try
STAT=`cat /usr/local/www/rec 2>/dev/null || echo 0`
maybe.