I have a bash script which runs a number of tasks on remote machines via SSH. Each script is quite long so I'd like to run them concurrently as background tasks. I also need for them all to complete before moving on. I know that I can use the wait command for the latter but when I stick & at the end to make it background task, it all stops working.
By stop working, I mean that the scripts don't seem to have run, but otherwise the main script still completes.
ssh root#machine1 'bash -s' < script1 my_parameter
ssh root#machine2 'bash -s' < script2 my_parameter
ssh root#machine3 'bash -s' < script3 my_parameter
wait
some_other_task
This works fine for me with Ubuntu 11.04:
#!/bin/bash
ssh user#server1 <command.sh &
ssh user#server2 <command.sh &
ssh user#server3 <command.sh &
wait
echo end
Content of command.sh:
sleep 5
hostname
Update:
With commandline arguments it gets more complicated. You can use a "here-document" and set to set $1 $2 $3.
#!/bin/bash
ssh -T root#server1 << EOF
#!/bin/bash
set -- $my_parameter1 $my_parameter2 $my_parameter3
$(cat script1)
EOF
ssh -T root#server2 << EOF
#!/bin/bash
set -- $my_parameter1 $my_parameter2 $my_parameter3
$(cat script2)
EOF
ssh -T root#server3 << EOF
#!/bin/bash
set -- $my_parameter1 $my_parameter2 $my_parameter3
$(cat script3)
EOF
wait
echo end
Related
How to drop ssh session in an automation script write in bash?
I have a script running local to trigger a script on a remote machine, and the script on remote machine will trigger another script running in the background...
I want to drop the session while keep the remote machine still running the background script, so I use nohup.
I have a local script localScript as follows
#!/bin/bash
echo "start remote trigger script..."
./trigger
the trigger script is ready on my remote machine with the following lines:
#!/bin/bash
echo "start script test..."
nohup ./test > output &
echo "start test script in background, exit..."
exit
The test script is a basic sleep loop just for testing...
#!/bin/bash
c=1
while [ "$c" -le 10 ]
do
echo "sleep 10 seconds, c=$c"
sleep 10s
c=$((c+1))
if [ "$c" -eq 10 ]
then
echo "max count reach, exit"
exit
fi
done
But what I found is the ssh keeps session alive (wait idle for 100 seconds), how can I drop the session?
The command I use is
sshpass -p XXXX ssh -o StrictHostKeyChecking=no user#IP 'bash -s' < localScript
This should work if you force pseudo-terminal allocation (-t):
sshpass -p XXXX ssh -t -o StrictHostKeyChecking=no user#IP 'bash -s' < localScript
I'd like to execute several commands in sequence on a remote machine, and some of the later commands depend on earlier ones. In the simplest possible example I get this:
ssh my_server "echo this is my_server; abc=2;"
this is my_server
abc=2: Command not found.
ssh my_server "echo this is my_server; abc=2; echo abc is $abc"
abc: undefined variable
For a bit of background info, what I actually want to do is piece together a path and launch a java application:
ssh my_server 'nohup sh -c "( ( echo this is my_server; jabref_exe=`which jabref`; jabref_dir=`dirname $jabref_exe`; java -jar $jabref_dir/../jabref.jar` $1 &/dev/null ) & )"' &
jabref_dir: Undefined variable.
That way, whenever jabref gets updated to a new version on the server, I won't have to manually update the path to the jar file. The jabref executable doesn't take arguments, but launching it with java -jar does, which is why I have to juggle the path a bit.
At the moment I have the list of commands in a separate script file and call
ssh my_server 'nohup sh -c "( ( my_script.sh &/dev/null ) & )"' &
which works, but since the ssh call is already inside one script file it would be nice to have everything together.
In this example
ssh my_server "echo this is my_server; abc=2;"
abc is set on the remote side, so it should be clear why it is not set on your local machine.
In the next example,
ssh my_server "echo this is my_server; abc=2; echo abc is $abc"
your local shell tries to expand $abc in the argument before it is ever sent to the remote host. A slight modification would work as you expected:
ssh my_server 'echo this is my_server; abc=2; echo abc is $abc'
The single quotes prevent your local shell from trying to expand $abc, and so the literal text makes it to the remote host.
To finally address your real question, try this:
jabref_dir=$( ssh my_server 'jabref_exe=$(which jabref); jabref_dir=$(dirname $jabref_exe);
java -jar $jabref_dir/../jabref.jar > /dev/null; echo $jabref_dir' )
This will run the quoted string as a command on your remote server, and output exactly one string: $jabref_dir. That string is captured and stored in a variable on your local host.
With some inspiration from chepner, I now have a solution that works, but only when called from a bash shell or bash script. It doesn't work from tcsh.
ssh my_server "bash -c 'echo this is \$HOSTNAME; abc=2; echo abc is \$abc;'"
Based on this, the code below is a local script which runs jabref on a remote server (although with X-forwarding by default and passwordless authentication the user can't tell it's remote):
#!/bin/bash
if [ -f "$1" ]
then
fname_start=$(echo ${1:0:4})
if [ "$fname_start" = "/tmp" ]
then
scp $1 my_server:$1
ssh my_server "bash -c 'source load_module jdk; source load_module jabref; java_exe=\$(which java); jabref_exe=\$(which jabref); jabref_dir=\$(echo \${jabref_exe%/bin/jabref});eval \$(java -jar \$jabref_dir/jabref.jar $1)'" &
else
echo input argument must be a file in /tmp.
else
echo this function requires 1 argument
fi
and this is the 1-line script load_module, since modulecmd sets environment variables and I couldn't figure out how to do that without sourcing a script.
eval `/path/to/modulecmd bash load $1`;
I also looked at heredocs, inspired by How to use SSH to run a shell script on a remote machine? and http://tldp.org/LDP/abs/html/here-docs.html. The nice part is that it works even from tcsh. I got this working from the command line, but not inside a script. That's probably easy enough to fix, but I've got a solution now so I'm happy :-)
ssh my_server 'bash -s' << EOF
echo this is \$HOSTNAME; abc=2; echo abc is \$abc;
EOF
I want to start a process using nohup on a remote machine via ssh. The problem is how to get the PID of the process started with nohup, so the "process actually doing something", not some outer shell instance or the like. Also, I want to store stdout and stderr in files, but that is not the issue here...
Locally, it works flawlessly using
nohup sleep 30 > out 2> err < /dev/null & echo $!
It is echoing me the exact PID of the command "sleep 30", which I can also see using "top" or "ps aux|grep sleep".
But I'm having trouble doing it remotely via ssh. I tried something like
ssh remote_machine 'nohup bash -c "( ( sleep 30 ) & )" > out 2> err < /dev/null'
but I cannot figure out where to place the "echo $!" so that it is displayed in my local shell. It is always showing me wrong PIDs, for example the one of the "bash" instance etc.
Has somebody an idea how to solve this?
EDIT:
OK, the "bash -c" might not be needed here. Like Lotharyx pointed out, I get the right PID just fine using
ssh remote 'nohup sleep 30 > out 2> err < /dev/null & echo $!'
but then the problem is that if you substitute "sleep 30" with something that produces output, say, "echo Hello World!", that output does not end up in the file "out", neither on the local nor on remote side. Anybody got an idea why?
EDIT2: My fault! There was just no space left on the other device, that's why the files "out" and "err" stayed empty!
So this is working. In addition, if one wants to call multiple commands in a row, separated by a semicolon (;), one can still use "bash -c", like so:
ssh remote 'nohup bash -c "echo bla;sleep 30;echo blupp" > out 2> err < /dev/null & echo $!'
Then it prints out the PID of the "bash -c" on the local side, which is just fine. (It is impossible to get the PID of the "innermost" or "busy" process, because every program itself can spawn new subprocesses, there is no way to find out...)
I tried the following (the local machine is Debian; the remote machine is CentOS), and it worked exactly as I think you're expecting:
~# ssh someone#somewhere 'nohup sleep 30 > out 2> err < /dev/null & echo $!'
someone#somewhere's password:
14193
~#
On the remote machine, I did ps -e, and saw this line:
14193 ? 00:00:00 sleep
So, clearly, on my local machine, the output is the PID of "sleep" executing on the remote machine.
Why are you adding bash to your command when sending it across an SSH tunnel?
I have a script I can run locally to remotely start a server:
#!/bin/bash
ssh user#host.com <<EOF
nohup /path/to/run.sh &
EOF
echo 'done'
After running nohup, it hangs. I have to hit ctrl-c to exit the script.
I've tried adding an explicit exit at the end of the here doc and using "-t" argument for ssh. Neither works. How do I make this script exit immediately?
EDIT: The client is OSX 10.6, server is Ubuntu.
I think the problem is that nohup can't redirect output when you come in from ssh, it only redirects to nohup.out when it thinks it's connected to a terminal, and I the stdin override you have will prevent that, even with -t.
A workaround might be to redirect the output yourself, then the ssh client can disconnect - it's not waiting for the stream to close. Something like:
nohup /path/to/run.sh > run.log &
(This worked for me in a simple test connecting to an Ubuntu server from an OS X client.)
The problem might be that ...
... ssh is respecting the POSIX standard when not closing the session
if a process is still attached to the tty.
Therefore a solution might be to detach the stdin of the nohup command from the tty:
nohup /path/to/run.sh </dev/null &
See: SSH Hangs On Exit When Using nohup
Yet another approach might be to use ssh -t -t to force pseudo-tty allocation even if stdin isn't a terminal.
man ssh | less -Ip 'multiple -t'
ssh -t -t user#host.com <<EOF
nohup /path/to/run.sh &
EOF
See: BASH spawn subshell for SSH and continue with program flow
Redirecting the stdin of the remote host from a here document while invoking ssh without an explicit command leads to the message: Pseudo-terminal will not be allocated because stdin is not a terminal.
To avoid this message either use ssh's -T switch to tell the remote host there is no need to allocate a pseudo-terminal or explicitly specify a command (such as /bin/sh) for the remote host to execute the commands provided by the here document.
If an explicit command is given to ssh, the default is to provide no login shell in the form of a pseudo-terminal, i. e. there will be no normal login session when a command is specified (see man ssh).
Without a command specified for ssh, on the other hand, the default is to create a pseudo-tty for an interactive login session on the remote host.
- ssh user#host.com <<EOF
+ ssh -T user#host.com <<EOF
+ ssh user#host.com /bin/bash <<EOF
As a rule, ssh -t or even ssh -t -t should only be used if there are commands that expect stdin / stdout to be a terminal (such as top or vim) or if it is necessary to kill the remote shell and its children when the ssh client command finishes execution (see: ssh command unexpectedly continues on other system after ssh terminates).
As far as I can tell, the only way to combine an ssh command that does not allocate a pseudo-tty and a nohup command that writes to nohup.out on the remote host is to let the nohup command execute in a pseudo-terminal not created by the ssh mechanism. This can be done with the script command, for example, and will avoid the tcgetattr: Inappropriate ioctl for device message.
#!/bin/bash
ssh localhost /bin/sh <<EOF
#0<&- script -q /dev/null nohup sleep 10 1>&- &
#0<&- script -q -c "nohup sh -c 'date; sleep 10 1>&- &'" /dev/null # Linux
0<&- script -q /dev/null nohup sh -c 'date; sleep 10 1>&- &' # FreeBSD, Mac OS X
cat nohup.out
exit 0
EOF
echo 'done'
exit 0
You need to add a exit 0 at the end.
I'd like to execute several commands in sequence on a remote machine, and some of the later commands depend on earlier ones. In the simplest possible example I get this:
ssh my_server "echo this is my_server; abc=2;"
this is my_server
abc=2: Command not found.
ssh my_server "echo this is my_server; abc=2; echo abc is $abc"
abc: undefined variable
For a bit of background info, what I actually want to do is piece together a path and launch a java application:
ssh my_server 'nohup sh -c "( ( echo this is my_server; jabref_exe=`which jabref`; jabref_dir=`dirname $jabref_exe`; java -jar $jabref_dir/../jabref.jar` $1 &/dev/null ) & )"' &
jabref_dir: Undefined variable.
That way, whenever jabref gets updated to a new version on the server, I won't have to manually update the path to the jar file. The jabref executable doesn't take arguments, but launching it with java -jar does, which is why I have to juggle the path a bit.
At the moment I have the list of commands in a separate script file and call
ssh my_server 'nohup sh -c "( ( my_script.sh &/dev/null ) & )"' &
which works, but since the ssh call is already inside one script file it would be nice to have everything together.
In this example
ssh my_server "echo this is my_server; abc=2;"
abc is set on the remote side, so it should be clear why it is not set on your local machine.
In the next example,
ssh my_server "echo this is my_server; abc=2; echo abc is $abc"
your local shell tries to expand $abc in the argument before it is ever sent to the remote host. A slight modification would work as you expected:
ssh my_server 'echo this is my_server; abc=2; echo abc is $abc'
The single quotes prevent your local shell from trying to expand $abc, and so the literal text makes it to the remote host.
To finally address your real question, try this:
jabref_dir=$( ssh my_server 'jabref_exe=$(which jabref); jabref_dir=$(dirname $jabref_exe);
java -jar $jabref_dir/../jabref.jar > /dev/null; echo $jabref_dir' )
This will run the quoted string as a command on your remote server, and output exactly one string: $jabref_dir. That string is captured and stored in a variable on your local host.
With some inspiration from chepner, I now have a solution that works, but only when called from a bash shell or bash script. It doesn't work from tcsh.
ssh my_server "bash -c 'echo this is \$HOSTNAME; abc=2; echo abc is \$abc;'"
Based on this, the code below is a local script which runs jabref on a remote server (although with X-forwarding by default and passwordless authentication the user can't tell it's remote):
#!/bin/bash
if [ -f "$1" ]
then
fname_start=$(echo ${1:0:4})
if [ "$fname_start" = "/tmp" ]
then
scp $1 my_server:$1
ssh my_server "bash -c 'source load_module jdk; source load_module jabref; java_exe=\$(which java); jabref_exe=\$(which jabref); jabref_dir=\$(echo \${jabref_exe%/bin/jabref});eval \$(java -jar \$jabref_dir/jabref.jar $1)'" &
else
echo input argument must be a file in /tmp.
else
echo this function requires 1 argument
fi
and this is the 1-line script load_module, since modulecmd sets environment variables and I couldn't figure out how to do that without sourcing a script.
eval `/path/to/modulecmd bash load $1`;
I also looked at heredocs, inspired by How to use SSH to run a shell script on a remote machine? and http://tldp.org/LDP/abs/html/here-docs.html. The nice part is that it works even from tcsh. I got this working from the command line, but not inside a script. That's probably easy enough to fix, but I've got a solution now so I'm happy :-)
ssh my_server 'bash -s' << EOF
echo this is \$HOSTNAME; abc=2; echo abc is \$abc;
EOF