run shell script on multiple remote machines - bash

Question

WARNING: newbie with bash shell scripting.
I've created the script to connect to multiple remote machines, one by one, check if a certain file has certain text already in it, if it does, move to the next machine and make the same check, if not, append the text to the file, then move to the next machine.
Currently, the script connects to the first remote machine but then does nothing when it connects. If I type exit to close the remote machine's connection, it then continues running the script, which does me no good because I'm not connected to the remote machine any longer.
on a sidenote, I'm not even sure if the rest of the code is correct, so please let me know if there are any glaring mistakes. This is actually my first attempt at writing a shell script from scratch.
#!/bin/bash
REMOTE_IDS=( root#CENSOREDIPADDRESS1
root#CENSOREDIPADDRESS2
root#CENSOREDIPADDRESS3
)
for REMOTE in "{$REMOTE_IDS[#]}"
do
ssh -oStrictHostKeyChecking=no $REMOTE_IDS
if grep LogDNAFormat "/etc/syslog-ng/syslog-ng.conf"
then
echo $REMOTE
echo "syslog-ng already modified. Skipping."
exit
echo -
else
echo $REMOTE
echo "Modifiying..."
echo "\n" >> syslog-ng.conf
echo "### START syslog-ng LogDNA Logging Directives ###" >> syslog-ng.conf
echo "template LogDNAFormat { template(\"<key:CENSOREDKEY> <${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} - $MSG\n\");" >> syslog-ng.conf
echo "template_escape(no);" >> syslog-ng.conf
echo "};" >> syslog-ng.conf
echo "destination d_logdna {" >> syslog-ng.conf
echo "udp(\"syslog-a.logdna.com\" port(CENSOREDPORT)" >> syslog-ng.conf
echo "template(LogDNAFormat));" >> syslog-ng.conf
echo "};" >> syslog-ng.conf
echo "log {" >> syslog-ng.conf
echo "source(s_src);" >> syslog-ng.conf
echo "destination(d_logdna);" >> syslog-ng.conf
echo "};" >> syslog-ng.conf
echo "### END syslog-ng LogDNA logging directives ###" >> syslog-ng.conf
killall -s 9 syslog-ng
sleep 5
/etc/init.d/syslog start
echo -
fi
done

Answer 1

Great question: Automating procedures via ssh is a laudable goal.
Let's start off with the first error in your code:
ssh -oStrictHostKeyChecking=no $REMOTE_IDS
should be:
ssh -oStrictHostKeyChecking=no $REMOTE
But that won't do everything either. If you want to ssh to run a set of commands, you can, but you'll need to pass those commands in a string as an argment to ssh.
ssh -oStrictHostKeyChecking=no $REMOTE 'Lots of code goes here - newlines ok'
For that to work, you'll need to have passwordless ssh configured ( or you'll be prompted for credentials ). This is covered in steps 1) and 2) in Alexei Grochev's post. One option for passwordless logins is to put public keys on the hosts you want to manage and, if necessary, change the IdentityFile in your local ~/.ssh/config ( you may not need to do this if you are using a default public / private key pair ) .
You've got to be careful with ssh stealing your stdin ( I don't think you'll have a problem in your case ). In the cases that you suspect that the ssh command is reading all your stdin input, you'll need to supply the -n parameter to ssh ( again, I think your code does not suffer from this problem, but I didn't look to carefully ).
I agree with tadman's comment, that this is a good application for Ansible. However, I wouldn't learn Ansible for this task alone. If you intend on doing a lot of remote automation, Ansible would be well worth your time learning and applying to this problem.
What I would suggest is pssh and pscp. These tools are awesome and take care of the "for" loop for you. They also perform the ssh calls in parallel and collect the results.
Here are the steps I would recommend:
1) Install pssh (pscp comes along for the ride).
2) Write your bash program as a separate file. It's so much easier to debug and update , etc. if your program isn't in a bunch of echo statements. Those hurt. Even my original suggestion of ssh user#host 'long string of commands' is difficult to debug. Just create a program file that runs on the remote hosts and debug it on the remote host ( as you can ) .
3) Now go back to your control host ( with that bash program ). Push it to all of the hosts under management with pscp. The syntax is as follows:
# Your bash program is at <local-file-path>
chmod +x <local-file-path>
pscp -h<hosts-file> -l root <local-file-path> <remote-file-path>
The -h option specifies a lists of hosts. So the would look like this:
CENSOREDIPADDRESS1
CENSOREDIPADDRESS2
CENSOREDIPADDRESS3
Incidentally, if you did not set up your public/private keys, you can specify the -A parameter and pscp and pssh will ask you for the root user's password. This isn't great for automation, but if you are doing a one time task it is a lot easier than setting your public/private keys.
4) Now execute that program on the remote hosts:
pssh -h<hosts-file> -i <remote-file-path>
The -i parameter tells pssh to wait for the program to execute on all hosts and return the stdout and stderr results in line.
In summary, pssh/pscp are GREAT for small tasks like this. For larger tasks, consider Ansible ( it basically works by sending python scripts over ssh and executing them remotely ). Puppet/Chef are way overkill for this, but they are fantastic tools for keeping your data center in the state that you want it in.

Answer 2

you can do this with puppet/chef.
but this can also be done with bash if you have patience. I don't want to give actual code because I think its best to understand logic first.
however since you asked, here is the flow you should follow:
make sure you have keys setup for all machines
create config with all the servers
put all servers into an array
create a loop to call each box and run your script (before you will have to scp the script to the home dir on the box so make sure it good to run)
you can also do what you want better imho and that's how I've done it before.
1) make a script to read your file and put it on cron to run every minute or whatever time is best, say echo out #size of file to a log file
2) all servers will have those scripts running so now you just run your script to fetch the data across all servers (iterate through your array of servers in your config file)
^^ that right there can also be done with php where you have an instance of a webserver reading the file. you can also create a web server with bash...since its only for 1 task its not terribly insane.
have fun.