This question already has answers here:
How to pass argument with exclamation mark on Linux?
(3 answers)
Closed 3 years ago.
I am not able to store any PID in a file on the remote machine when running a script in background through ssh.
I need to store the PID of the script process in a file in purpose to kill it whenever needed. When running the exact command on the remote machine it is working, why through ssh it is not working so ?
What is wrong with the following command:
ssh user#remote_machine "nohup ./script.sh > /dev/null 2>&1 & echo $! > ./pid.log"
Result: The file pid.log is created but empty.
Expected: The file pid.log should contain the PID of the running script.
Use
ssh user#remote_machine 'nohup ./script.sh > /dev/null 2>&1 & echo $! > ./pid.log'
OR
ssh user#remote_machine "nohup ./script.sh > /dev/null 2>&1 & echo \$! > ./pid.log"
Issue:
Your $! was getting expanded locally, before calling ssh at all.
Worse, before calling the ssh command, if there was a process stared in the background, then $! would have expanded to that and complete ssh command would have got expanded to contain that PID as argument to echo.
e.g.
$ ls &
[12342] <~~~~ This is the PID of ls
$ <~~~~ Prompt returns immediately because ls was stared in background.
myfile1 myfile2 <~~~~ Output of ls.
[1]+ Done ls
#### At this point, $! contains 12342
$ ssh user#remote "command & echo $! > pidfile"
# before even calling ssh, shell internally expands it to:
$ ssh user#remote "command & echo 12342 > pidfile"
And it will put the wrong PID in the pidfile.
I'm creating a simple Traffic simulator: a client that curl a Webserver every 10 seconds. The client(Debian) and the WebServer are configured using Ansible. The background-cycle is closed when the Ansible-SSH connection is closed.
At first I launched the command:
$ while true; do curl python_webserver:8000; sleep10; done </dev/null >/dev/null 2>&1 &;
$ disown
And It works fine from Bash, but if I put it into a script, it exit at ssh-connection end.
I try some other solutions, like using:
$ nohup [command]
$ nohup /bin/bash -c '[command]
or using "deamonize" but nothing worked. Nothing I found online works; maybe I'm missing something important. (Writing the pid is important, but not fundamental)
I here the script, maybe there is a big newbie error.
#!/bin/bash
PORT=8000
while true; do
curl python_webserver:$PORT
sleep 10
done >/dev/null 2>&1 &
ANSIBLE_CLIENT_PID=$!
echo $ANSIBLE_CLIENT_PID >> /tmp/ANSIBLE_PID.txt
disown
The answers is
nohup sh -c 'while true; do curl python_webserver:'"$PORT"'; sleep 10; done >/dev/null 2>&1' &
Probably I made a mistake with quotation marks.
Thank you all for you help
(especially Kamil Cuk for the solution)
I want to have one script which starts a services in another server.
I have tested that the script works as expected in the server where the server is going to run.
This is the code which starts the service and monitors the log until it is in the startup process:
pkill -f "$1"
nohup java -jar -Dspring.profiles.active=$PROFILE $1 &
tail -n 0 -f nohup.out | while read LOGLINE
do
echo $LOGLINE
[[ "${LOGLINE}" == *"$L_LOG_STRING"* ]] && pkill -P $$ tail
done
This works fine as long as I execute that from that machine.
Now I want to call that script from another server:
#!/usr/bin/env bash
DESTINATION_SERVER=$1
ssh root#$DESTINATION_SERVER /bin/bash << EOF
echo "Restarting first service..."
/usr/local/starter.sh -s parameter
echo "Restarting second service..."
/usr/local/starter.sh -s parameter2
EOF
Well, everytime I try that the script of the remote server gets stuck in the "while READ" loop. But as I said, when I execute it locally from the server works fine, and in my "not simplified script" I´m not using any system variable or similar.
Update: I just tried to simplify the code even more with the following lines in the first scenario:
pkill -f "$1"
nohup java -jar -Dspring.profiles.active=$PROFILE $1 &
tail -n 0 -f nohup.out | sed "/$L_LOG_STRING/ q"
I'd say the problem is some how in the "|" through ssh, but I still can find why.
it seems that the problem comes from not having an interactive console when you execute the ssh command, therefore the nohup command behaves strangly.
I could solve it in two ways, outputing the code to the file explicitly:
"nohup java -jar -Dspring.profiles.active=test &1 >> nohup.out &"
instead of:
"nohup java -jar -Dspring.profiles.active=test &1&"
Or changing the way I access via ssh adding the tt option (just one did not work):
ssh -tt root#$DESTINATION_SERVER /bin/bash << EOF
But this last solution could lead to other problems with some character, so unless someone suggests another solution that is my patch which makes it work.
So I have a problem similar to how to send ssh job to background.
I have a windows c# program automated to execute tcpdump on a remote linux os using http://sshnet.codeplex.com/. I'm trying to execute tcpdump on the remote linux and leave it running after I disconnect.
I've been doing a lot of debugging using plink, but cannot seem to achieve the desired result. I've tried:
plink root#10.5.1.1 bash -c "tcpdump -i eth0 -w test.cap"
but it holds the sshclient until I ctrl+C (not going to work for automated solution). I've also tried variations of:
plink root#10.5.1.1 bash -c "tcpdump -i eth0 -w test.cap &"
but either the command is not executed at all (test.cap does not exist) or is terminated immediately (test.cap contains 1 line). During testing, I've left a ping going, so the capture should have somthing...
The previously mentioned link solves the problem with screen, but the remote linux os is not configurable and does not have screen. Any suggestions are welcome.
In the latter case, your tcpdump process is probably being aborted when you disconnect. Try:
plink root#10.5.1.1 bash -c "nohup tcpdump -i eth0 -w test.cap &"
See the manpage for nohup. You may also want to consider redirecting stdout and stderr to a file or /dev/null to prevent nohup from writing output to a file:
plink root#10.5.1.1 bash -c "nohup tcpdump -i eth0 -w test.cap >/dev/null 2>&1 &"
I had a similar problem while starting a remote application. This pattern worked for me on Debian servers:
ssh root#server "nohup /usr/local/bin/app -c cfg &; exit"
addition: for another test the above didn't work, ie. the command didn't start on the remote server. Adding a command that returns successfully before the exit seems to work.
ssh root#server "nohup /usr/local/bin/otherapp &; w; exit"
I had a similar situation:
(on windows machine) i wanted to create a ms batch script to open an SSH connection to a raspberry pi and execute a local script in the background.
I found that combining both Raj's and fahd's answers did the trick for me:
my ms batch script:
plink -load "raspberry Pi" -t -m startCommand.txt
the content of startCommand.txt is as follows:
nohup /home/pi/myscript >/dev/null 2>&1 &
w
exit
The ">/dev/null 2>&1 " is important!
I found out (the hard way) that the RPi's SDcard kept getting full by an extremely large nohup.out file (and with a full SDcard, the RPi couldn't even login properly)
reasoning:
I used the -load to load a saved session in PuTTY (i do this because i am authenticating with public/private keys instead of passwords, but this should be the same as simply typing in the host)
then -t (as recommended by Raj)
then -m to load a list of commands in that file
without the parameter "-t" and without the "w" and "exit", my batch script would just run, not execute 'myscript' and close again.
I had the same issue. I had a scrip in which I had nohup tcpdump .... & . I could not use ssh to run it as it dies when the ssh finished. The solution I came up with was super simple. I just added sleep 5 to the end of my script and it works just fine. It seems tcpdump needs some seconds to go to background safely before you exit even with nohup.
I had the same problem, and I found that the "-t" option seems to be important to nohup. I found the nohup wasn't taking affect without the "-t" option.
ssh -t user#remote 'nohup tcpdump -i any -w /tmp/somefile &>/dev/null & sleep 2'
I think that I've nailed it, at least in IBM AIX
I'm using
ssh -tq user#host "/path/start-tcpdump.ksh"
(authentication is done by publick key).
I was having inconsistent results using simple "nohup tcpdump .... &", sometimes it worked, sometimes it did not, sometimes it even blocked and I had to disconnect the session.
So far, this is working ok, I can't really say WHY it is working, but it is...
This is my start-tcpip.ksh
#!/usr/bin/ksh
HOST=$(uname -n)
FILTER="port not 22"
(tcpdump -i en1 -w $HOST-en1.cap $FILTER >/dev/null 2>&1 ) &
sleep 2
(tcpdump -i en2 -w $HOST-en2.cap $FILTER >/dev/null 2>&1 ) &
sleep 2
exit 0
This is a follow-on question to the How do you use ssh in a shell script? question. If I want to execute a command on the remote machine that runs in the background on that machine, how do I get the ssh command to return? When I try to just include the ampersand (&) at the end of the command it just hangs. The exact form of the command looks like this:
ssh user#target "cd /some/directory; program-to-execute &"
Any ideas? One thing to note is that logins to the target machine always produce a text banner and I have SSH keys set up so no password is required.
I had this problem in a program I wrote a year ago -- turns out the answer is rather complicated. You'll need to use nohup as well as output redirection, as explained in the wikipedia artcle on nohup, copied here for your convenience.
Nohuping backgrounded jobs is for
example useful when logged in via SSH,
since backgrounded jobs can cause the
shell to hang on logout due to a race
condition [2]. This problem can also
be overcome by redirecting all three
I/O streams:
nohup myprogram > foo.out 2> foo.err < /dev/null &
This has been the cleanest way to do it for me:-
ssh -n -f user#host "sh -c 'cd /whereever; nohup ./whatever > /dev/null 2>&1 &'"
The only thing running after this is the actual command on the remote machine
Redirect fd's
Output needs to be redirected with &>/dev/null which redirects both stderr and stdout to /dev/null and is a synonym of >/dev/null 2>/dev/null or >/dev/null 2>&1.
Parantheses
The best way is to use sh -c '( ( command ) & )' where command is anything.
ssh askapache 'sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'
Nohup Shell
You can also use nohup directly to launch the shell:
ssh askapache 'nohup sh -c "( ( chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'
Nice Launch
Another trick is to use nice to launch the command/shell:
ssh askapache 'nice -n 19 sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'
If you don't/can't keep the connection open you could use screen, if you have the rights to install it.
user#localhost $ screen -t remote-command
user#localhost $ ssh user#target # now inside of a screen session
user#remotehost $ cd /some/directory; program-to-execute &
To detach the screen session: ctrl-a d
To list screen sessions:
screen -ls
To reattach a session:
screen -d -r remote-command
Note that screen can also create multiple shells within each session. A similar effect can be achieved with tmux.
user#localhost $ tmux
user#localhost $ ssh user#target # now inside of a tmux session
user#remotehost $ cd /some/directory; program-to-execute &
To detach the tmux session: ctrl-b d
To list screen sessions:
tmux list-sessions
To reattach a session:
tmux attach <session number>
The default tmux control key, 'ctrl-b', is somewhat difficult to use but there are several example tmux configs that ship with tmux that you can try.
I just wanted to show a working example that you can cut and paste:
ssh REMOTE "sh -c \"(nohup sleep 30; touch nohup-exit) > /dev/null &\""
You can do this without nohup:
ssh user#host 'myprogram >out.log 2>err.log &'
Quickest and easiest way is to use the 'at' command:
ssh user#target "at now -f /home/foo.sh"
I think you'll have to combine a couple of these answers to get what you want. If you use nohup in conjunction with the semicolon, and wrap the whole thing in quotes, then you get:
ssh user#target "cd /some/directory; nohup myprogram > foo.out 2> foo.err < /dev/null"
which seems to work for me. With nohup, you don't need to append the & to the command to be run. Also, if you don't need to read any of the output of the command, you can use
ssh user#target "cd /some/directory; nohup myprogram > /dev/null 2>&1"
to redirect all output to /dev/null.
This worked for me may times:
ssh -x remoteServer "cd yourRemoteDir; ./yourRemoteScript.sh </dev/null >/dev/null 2>&1 & "
You can do it like this...
sudo /home/script.sh -opt1 > /tmp/script.out &
It appeared quite convenient for me to have a remote tmux session using the tmux new -d <shell cmd> syntax like this:
ssh someone#elsewhere 'tmux new -d sleep 600'
This will launch new session on elsewhere host and ssh command on local machine will return to shell almost instantly. You can then ssh to the remote host and tmux attach to that session. Note that there's nothing about local tmux running, only remote!
Also, if you want your session to persist after the job is done, simply add a shell launcher after your command, but don't forget to enclose in quotes:
ssh someone#elsewhere 'tmux new -d "~/myscript.sh; bash"'
Actually, whenever I need to run a command on a remote machine that's complicated, I like to put the command in a script on the destination machine, and just run that script using ssh.
For example:
# simple_script.sh (located on remote server)
#!/bin/bash
cat /var/log/messages | grep <some value> | awk -F " " '{print $8}'
And then I just run this command on the source machine:
ssh user#ip "/path/to/simple_script.sh"
If you run remote command without allocating tty, redirect stdout/stderr works, nohup is not necessary.
ssh user#host 'background command &>/dev/null &'
If you use -t to allocate tty to run interactive command along with background command, and background command is the last command, like this:
ssh -t user#host 'bash -c "interactive command; nohup backgroud command &>/dev/null &"'
It's possible that background command doesn't actually start. There's race here:
bash exits after nohup starts. As a session leader, bash exit results in HUP signal sent to nohup process.
nohup ignores HUP signal.
If 1 completes before 2, the nohup process will exit and won't start the background command at all. We need to wait nohup start the background command. A simple workaroung is to just add a sleep:
ssh -t user#host 'bash -c "interactive command; nohup backgroud command &>/dev/null & sleep 1"'
The question was asked and answered years ago, I don't know if openssh behavior changed since then. I was testing on:
OpenSSH_8.6p1, OpenSSL 1.1.1g FIPS 21 Apr 2020
I was trying to do the same thing, but with the added complexity that I was trying to do it from Java. So on one machine running java, I was trying to run a script on another machine, in the background (with nohup).
From the command line, here is what worked: (you may not need the "-i keyFile" if you don't need it to ssh to the host)
ssh -i keyFile user#host bash -c "\"nohup ./script arg1 arg2 > output.txt 2>&1 &\""
Note that to my command line, there is one argument after the "-c", which is all in quotes. But for it to work on the other end, it still needs the quotes, so I had to put escaped quotes within it.
From java, here is what worked:
ProcessBuilder b = new ProcessBuilder("ssh", "-i", "keyFile", "bash", "-c",
"\"nohup ./script arg1 arg2 > output.txt 2>&1 &\"");
Process process = b.start();
// then read from process.getInputStream() and close it.
It took a bit of trial & error to get this working, but it seems to work well now.
YOUR-COMMAND &> YOUR-LOG.log &
This should run the command and assign a process id you can simply tail -f YOUR-LOG.log to see results written to it as they happen. you can log out anytime and the process will carry on
If you are using zsh then use program-to-execute &! is a zsh-specific shortcut to both background and disown the process, such that exiting the shell will leave it running.
A follow-on to #cmcginty's concise working example which also shows how to alternatively wrap the outer command in double quotes. This is how the template would look if invoked from within a PowerShell script (which can only interpolate variables from within double-quotes and ignores any variable expansion when wrapped in single quotes):
ssh user#server "sh -c `"($cmd) &>/dev/null </dev/null &`""
Inner double-quotes are escaped with back-tick instead of backslash. This allows $cmd to be composed by the PowerShell script, e.g. for deployment scripts and automation and the like. $cmd can even contain a multi-line heredoc if composed with unix LF.
First follow this procedure:
Log in on A as user a and generate a pair of authentication keys. Do not enter a passphrase:
a#A:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/a/.ssh/id_rsa):
Created directory '/home/a/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/a/.ssh/id_rsa.
Your public key has been saved in /home/a/.ssh/id_rsa.pub.
The key fingerprint is:
3e:4f:05:79:3a:9f:96:7c:3b:ad:e9:58:37:bc:37:e4 a#A
Now use ssh to create a directory ~/.ssh as user b on B. (The directory may already exist, which is fine):
a#A:~> ssh b#B mkdir -p .ssh
b#B's password:
Finally append a's new public key to b#B:.ssh/authorized_keys and enter b's password one last time:
a#A:~> cat .ssh/id_rsa.pub | ssh b#B 'cat >> .ssh/authorized_keys'
b#B's password:
From now on you can log into B as b from A as a without password:
a#A:~> ssh b#B
then this will work without entering a password
ssh b#B "cd /some/directory; program-to-execute &"
I think this is what you need:
At first you need to install sshpass on your machine.
then you can write your own script:
while read pass port user ip; do
sshpass -p$pass ssh -p $port $user#$ip <<ENDSSH1
COMMAND 1
.
.
.
COMMAND n
ENDSSH1
done <<____HERE
PASS PORT USER IP
. . . .
. . . .
. . . .
PASS PORT USER IP
____HERE