I have a bash script that ssh in to different machines in a loop, this script also reboots the machines if there is a reboot=y parameter passed (using jenkins).
Issue -
If reboot = y, it fails to loop the ssh command for each host one by one, then echo "rebooting host" then finally run sh app-services.sh $APP
My goal is - after reboot is done I want it to ping the hosts one by one and if ping returns success, ssh back into the machine and restart the services of the application hosted.
My current script -
#!/bin/bash
IP_S="${HOSTS_STAGE}"
IP_P="${HOSTS_PROD}"
APP="$APPLICATION"
export APP
export host
if [ "${ENVIRONMENT}" == 'stage' ]; then
IP="${IP_S}"
elif [ "${ENVIRONMENT}" == 'prod' ]; then
IP="${IP_P}"
fi
echo "${IP}"
for host in ${IP}; do
echo "########################################"
echo "## Running yum update on ${host}"
echo "########################################"
if [ "${REBOOT}" == 'n' ]; then
ssh -i "${id_rsa}" -o StrictHostKeyChecking=no dvopsadmin#${host} "sudo yum update -y"
echo "## Skipping reboot for ${host}"
else
ssh -i "${id_rsa}" -o StrictHostKeyChecking=no dvopsadmin#${host[#]} "sudo reboot"
echo "## Rebooting ${host}"
ping -w 30 -c 1 ${host}
# This is the other script that restart app services
sh app-services.sh $APP
fi
done
Related
How to drop ssh session in an automation script write in bash?
I have a script running local to trigger a script on a remote machine, and the script on remote machine will trigger another script running in the background...
I want to drop the session while keep the remote machine still running the background script, so I use nohup.
I have a local script localScript as follows
#!/bin/bash
echo "start remote trigger script..."
./trigger
the trigger script is ready on my remote machine with the following lines:
#!/bin/bash
echo "start script test..."
nohup ./test > output &
echo "start test script in background, exit..."
exit
The test script is a basic sleep loop just for testing...
#!/bin/bash
c=1
while [ "$c" -le 10 ]
do
echo "sleep 10 seconds, c=$c"
sleep 10s
c=$((c+1))
if [ "$c" -eq 10 ]
then
echo "max count reach, exit"
exit
fi
done
But what I found is the ssh keeps session alive (wait idle for 100 seconds), how can I drop the session?
The command I use is
sshpass -p XXXX ssh -o StrictHostKeyChecking=no user#IP 'bash -s' < localScript
This should work if you force pseudo-terminal allocation (-t):
sshpass -p XXXX ssh -t -o StrictHostKeyChecking=no user#IP 'bash -s' < localScript
So i am running this script to check if a java server is up remotely by sshing into remote. If it is down, i am trying to exit and run another script locally. However, after the exit command, it is still in the remote directory.
ssh -i ec2-user#$DNS << EOF
if ! lsof -i | grep -q java ; then
echo "java server stopped running"
# want to exit ssh
exit
# after here when i check it is still in ssh
# I want to run another script locally in the same directory as the current script
./other_script.sh
else
echo "java server up"
fi;
EOF
The exit is exiting the ssh session and so never gets to the execution of the other_script.sh line in the HEREDOC. It would be better to place this outside of the script and actioned from the exit status of the HEREDOC/ssh and so:
ssh -i ec2-user#$DNS << EOF
if ! lsof -i | grep -q java ; then
echo "java server stopped running"
exit 7 # Set the exit status to a number that isn't standard in case ssh fails
else
echo "java server up"
fi;
EOF
if [[ $? -eq 7 ]]
then
./other_script.sh
fi
Im trying to check connections for a list of servers. I want to loop through the list, check if a connection works and if yes, do some stuff, if not, echo out a problem message.
My problem is:
the script stops at the first node without echoing the $?.
So, whats wrong with my for-loop?
These vars are included from a config file:
$nodes is a list of server IPs like 1.1.1.1,2.2.2.2,10.10.10.10
$user is one string
for node in $(echo $nodes | sed "s/,/ /g")
do
echo "Checking Node: $node"
ssh -q -o ConnectTimeout=3 $user#$node echo ok
echo $?
if [[ $? != 0 ]]
then
echo "Problem in logging into $node"
else
# do some stuff here
fi
done
EDIT #1:
for node in $(echo $nodes | sed "s/,/ /g")
do
echo "Checking Node: $node"
ssh -q -t -o ConnectTimeout=3 $user#$node "echo ok"
retcode=$?
echo $retcode
if [[ "$retcode" -ne 0 ]]
then
echo "Problem in logging into $node"
else
echo "OK"
fi
done
It is because ssh first asks you to validate The authority of the host and If you accept the authority it will ask for password. That is why your command does not return to shell and waits for input.
If your intention is just validating ssh connection, then you may consider to use
telnet <your_host> <port> < /dev/null
But if your intend is to run some commands you need a trust relationship between hosts. In that case you can use:
Execute this commands:
ssh-keygen -t rsa
then
ssh-copy-id -i root#ip_address
Now you can connect with
ssh <user>#<host>
Furher information
You can add -tto make virtual terminal and add quotes on command:
ssh -q -t -o ConnectTimeout=3 ${user}#${node} "echo ok"
Also use -ne instead of != which is for compare strings
if [[ "$?" -ne 0 ]]
Also echo $? mess the return code. You should use something like:
ssh -q -t -o ConnectTimeout=3 ${user}#${node} "echo ok"
retcode=$?
echo $retcode
if [[ "$retcode" -ne 0 ]]
You can rewrite ssh command like this to avoid problems with ssh host keys
ssh -q -t -o StrictHostKeyChecking=no -o ConnectTimeout=3 ${user}#${node} "echo ok"
For instance, I wrote a simple script to delete all the jar files on remote machines.
## distribute commands to machines
for machine in $machines
do
echo $machine
ssh -o "StrictHostKeyChecking no" $machine "rm /home/ubuntu/test.jar" &
done
I want to know whether all these commands were successful. Is there any convenient way?
Thanks!
Get rid of the & backgrounding the sshes and then you can check their exit status with if or ||.
for machine in $machines; do
echo $machine
if ! ssh -o StrictHostKeyChecking=no $machine rm /home/ubuntu/test.jar; then
echo "didn't work" >&2
fi
done
or
succeeded=true
for machine in $machines; do
echo $machine
ssh -o StrictHostKeyChecking=no $machine rm /home/ubuntu/test.jar || succeeded=false
done
if ! $succeeded; then
echo "one or more removals failed" >&2
fi
Hi what is the best way to check to see if SSH fails for whatever reason?
Can I use a IF statement ( if it fails then do something)
I'm using the ssh command in a loop and passing my hosts names form a flat file.
so I do something like:
for i in `cat /tmp/hosts` ; do ssh $i 'hostname;sudo ethtool eth1'; done
I get sometime this error or I just cannot connect
ssh: host1 Temporary failure in name resolution
I want to skip the hosts that I cannot connect to is SSH fails. What is the best way to do this? Is there a runtime error I can trap to bypass the hosts that I cannot ssh into for whatever reason, perhaps ssh is not allowed or I do not have the right password ?
Thanking you in advance
Cheers
To check if there was a problem connecting and/or running the remote command:
if ! ssh host command
then
echo "SSH connection or remote command failed"
fi
To check if there was a problem connecting, regardless of success of the remote command (unless it happens to return status 255, which is rare):
if ssh host command; [ $? -eq 255 ]
then
echo "SSH connection failed"
fi
Applied to your example, this would be:
for i in `cat /tmp/hosts` ;
do
if ! ssh $i 'hostname;sudo ethtool eth1';
then
echo "Connection or remote command on $i failed";
fi
done
You can check the return value that ssh gives you as originally shown here:
How to create a bash script to check the SSH connection?
$ ssh -q user#downhost exit
$ echo $?
255
$ ssh -q user#uphost exit
$ echo $?
0
EDIT - I cheated and used nc
Something like this:
#!/bin/bash
ssh_port_is_open() { nc -z ${1:?hostname} 22 > /dev/null; }
for host in `cat /tmp/hosts` ; do
if ssh_port_is_open $host; then
ssh -o "BatchMode=yes" $i 'hostname; sudo ethtool eth1';
else
echo " $i Down"
fi
done