Bash script to set up a temporary SSH tunnel - bash

On Cygwin, I want a Bash script to:
Create an SSH tunnel to a remote server.
Do some work locally that uses the tunnel.
Then shut down the tunnel.
The shutdown part has me perplexed.
Currently, I have a lame solution. In one shell I run the following to create a tunnel:
# Create the tunnel - this works! It runs forever, until the shell is quit.
ssh -nNT -L 50000:localhost:3306 jm#sampledomain.com
Then, in another shell window, I do my work:
# Do some MySQL stuff over local port 50000 (which goes to remote port 3306)
Finally, when I am done, I close the first shell window to kill the tunnel.
I'd like to do this all in one script like:
# Create tunnel
# Do work
# Kill tunnel
How do I keep track of the tunnel process, so I know which one to kill?

You can do this cleanly with an ssh 'control socket'. To talk to an already-running SSH process and get it's pid, kill it etc. Use the 'control socket' (-M for master and -S for socket) as follows:
$ ssh -M -S my-ctrl-socket -fNT -L 50000:localhost:3306 jm#sampledomain.com
$ ssh -S my-ctrl-socket -O check jm#sampledomain.com
Master running (pid=3517)
$ ssh -S my-ctrl-socket -O exit jm#sampledomain.com
Exit request sent.
Note that my-ctrl-socket will be an actual file that is created.
I got this info from a very RTFM reply on the OpenSSH mailing list.

You can tell SSH to background itself with the -f option but you won't get the PID with $!.
Also instead of having your script sleep an arbitrary amount of time before you use the tunnel, you can use -o ExitOnForwardFailure=yes with -f and SSH will wait for all remote port forwards to be successfully established before placing itself in the background. You can grep the output of ps to get the PID. For example you can use
...
ssh -Cfo ExitOnForwardFailure=yes -N -L 9999:localhost:5900 $REMOTE_HOST
PID=$(pgrep -f 'N -L 9999:')
[ "$PID" ] || exit 1
...
and be pretty sure you're getting the desired PID

You can tell ssh to go into background with & and not create a shell on the other side (just open the tunnel) with a command line flag (I see you already did this with -N).
Save the PID with PID=$!
Do your stuff
kill $PID
EDIT: Fixed $? to $! and added the &

I prefer to launch a new shell for separate tasks and I often use the following command combination:
$ sudo bash; exit
or sometimes:
$ : > sensitive-temporary-data.txt; bash; rm -f sensitive-temporary-data.txt; exit
These commands create a nested shell where I can do all my work; when I'm finished I hit CTRL-D and the parent shell cleans up and exits as well. You could easily throw bash; into your ssh tunnel script just before the kill part so that when you log out of the nested shell your tunnel will be closed:
#!/bin/bash
ssh -nNT ... &
PID=$!
bash
kill $PID

You could launch the ssh with a & a the end, to put it in the background and grab its id when doing. Then you just have to do a kill of that id when you're done.

A simple bash script to solve your problem.
# Download then put in $PATH
wget https://raw.githubusercontent.com/ijortengab/bash/master/commands/command-keep-alive.sh
mv command-keep-alive.sh -t /usr/local/bin
# open tunnel, put script in background
command-keep-alive.sh "ssh -fN -o ServerAliveInterval=10 -o ServerAliveCountMax=2 -L 33306:localhost:3306 myserver" /tmp/my.pid &
# do something
mysql --port 33306
# close tunnel
kill $(cat /tmp/my.pid)

https://github.com/aronpc/remina-ssh-tunnel
#!/usr/bin/env sh
scriptname="$(basename $0)"
actionname="$1"
tunnelname=$(echo "$2" | iconv -t ascii//TRANSLIT | sed -E 's/[^a-zA-Z0-9-]+/-/g' | sed -E 's/^-+|-+$//g' | tr A-Z a-z)
remotedata="$3"
tunnelssh="$4"
if [ $# -lt 4 ]
then
echo "Usage: $scriptname start | stop LOCAL_PORT:RDP_IP:RDP_PORT SSH_NODE_IP"
exit
fi
case "$actionname" in
start)
echo "Starting tunnel to $tunnelssh"
ssh -M -S ~/.ssh/sockets/$tunnelname.control -fnNT -L $remotedata $tunnelssh
ssh -S ~/.ssh/sockets/$tunnelname.control -O check $tunnelssh
;;
stop)
echo "Stopping tunnel to $tunnelssh"
ssh -S ~/.ssh/sockets/$tunnelname.control -O exit $tunnelssh
;;
*)
echo "Did not understand your argument, please use start|stop"
;;
esac
usage example
Edit or create new remmina server connection
schema
~/.ssh/rdp-tunnel.sh ACTION TUNNELNAME LOCAL_PORT:REMOTE_SERVER:REMOTE_PORT TUNNEL_PROXY
name
description
ACTION
start|stop
TUNNELNAME
"string identify socket" slugify to create socket file into ~/.ssh/sockets/string-identify-socket.control
LOCAL_PORT
the door that will be exposed locally if we use the same port for two connections it will crash
REMOTE_SERVER
the ip of the server that you would access if you had it on the proxy server that will be used
REMOTE_PORT
the service port that runs on the server
TUNNEL_PROXY
the connection you are going to use as a proxy, it needs to be in your ~/.ssh/config preferably using the access keys
I use the combination (% g-% p) of the remmina group name and connection name to be my TUNNELNAME (this needs to be unique, it will see the socket name)
pre-command
~/.ssh/rdp-tunnel.sh start "%g-%p" 63394:192.168.8.176:3389 tunnel-name-1
post-command
~/.ssh/rdp-tunnel.sh stop "%g-%p" 63394:192.168.8.176:3389 tunnel-name-1
you can and should use this script to access anything, I use it constantly to access systems and services that do not have a public ip going through 1,2,3,4,5 or more ssh proxies
see more into :
ssh config
ssh mach
ssh jump hosts
sshuttle python ssh
Refs:
https://remmina.org/remmina-rdp-ssh-tunnel/
https://kgibran.wordpress.com/2019/03/13/remmina-rdp-ssh-tunnel-with-pre-and-post-scripts/
Bash script to set up a temporary SSH tunnel
https://gist.github.com/oneohthree/f528c7ae1e701ad990e6

Related

Unable to fully close remote SSH tunnel in script's exit

I'm writing a script, which double-hop SSH-forwards port 80 from our remotely deployed VMs, and opens this "status page" in a local browser. To open it, the SSH tunnel must be "backgrounded", however doing so causes the SSH tunnel to exit with a persistent tunnel remaining on the SSH server that I'm tunneling through (bastion). Here is the script, so far:
#!/bin/sh
# SSH needs a HUP when this script exits
shopt -s huponexit
echo "SSH Forwards the VM status page for a given host..."
read -p "Host Name: " CODE
PORT=$(($RANDOM + 1024))
# "-t -t" (force tty) needed to avoid orphan tunnels on bastion after exit. (Only seems to work when not backgrounded?)
ssh -t -t -4L $PORT:localhost:$PORT user1#bastion sudo ssh -4NL $PORT:localhost:80 root#$CODE.internal-vms &
PID=$!
# Open browser to VM Status Page
sleep 1
open http://localhost:$PORT/
# Runs the SSH tunnel in the background, ensuring it gets killed on shell's exit...
bash
kill -CONT $PID
#kill -QUIT $PID
echo "Killed SSH Tunnel. Exiting..."
sleep 2
Unfortunately, given the backgrounding of the SSH tunnel (using & on line 10), when the script is killed (via CTRL-C), the "bastion" server ends up having an orphaned SSH connection remaining indefinitely.
The "-t -t" and "shopt -s huponexit" are fixed I've tried, but don't seem to help. I've also tried various SIG's in the final kill. What am I doing wrong here? Thanks for the assistance!
The -f flag can be used to background the process. To end the connection, ssh -O exit user1#bastion is a better option than kill which is rather violent.
I would do it like this. Fyi, I didn't test the modified script, although I regularly use a similar, long SSH command.
#!/bin/sh
# SSH needs a HUP when this script exits
shopt -s huponexit
echo "SSH Forwards the VM status page for a given host..."
read -p "Host Name: " CODE
PORT=$(($RANDOM + 1024))
# "-t -t" (force tty) needed to avoid orphan tunnels on bastion after exit. (Only seems to work when not backgrounded?)
ssh -t -t -f -4L $PORT:localhost:$PORT user1#bastion sudo ssh -4NL $PORT:localhost:80 root#$CODE.internal-vms
#PID=$!
# Open browser to VM Status Page
sleep 1
open http://localhost:$PORT/
# Runs the SSH tunnel in the background, ensuring it gets killed on shell's exit...
#bash
#kill -CONT $PID
#kill -QUIT $PID
ssh -O exit user#bastion
echo "Killed SSH Tunnel. Exiting..."
sleep 2

Authenticating with user/password *once* for multiple commands? (session multiplexing)

I got this trick from Solaris documentation, for copying ssh public keys to remote hosts.
note: ssh-copy-id isn't available on Solaris
$ cat some_data_file | ssh user#host "cat >/tmp/some_data_file; some_shell_cmd"
I wanted to adapt it to do more involved things.
Specifically I wanted some_shell_command to be a script sent from the local host to execute on the remote host... a script would interact with the local keyboard (e.g. prompt user when the script was running on the remote host).
I experimented with ways of sending multiple things over stdin from multiple sources. But certain things that work in in local shell don't work over ssh, and some things, such as the following, didn't do what I wanted at all:
$ echo "abc" | cat <(echo "def") # echoes: def (I wanted abc\ndef)
$ echo "abc" | cat < <(echo "def") # echoes: def (I wanted abc\ndef)
$ echo "abc" | cat <<-EOF
> echo $(</dev/stdin) #echoes: echo abc (I wanted: abc)
> EOF
# messed with eval for the above but that was a problem too.
#chepner concluded it's not feasible to do all of that in a single ssh command. He suggested a theoretical alternative that didn't work as hoped, but I got it working after some research and tweaking and documented the results of that and posted it as an answer to this question.
Without that solution, having to run multiple ssh, and scp commands by default entails being prompted for password multiple times, which is a major drag.
I can't expect all the users of a script I write in a multi-user environment to configure public key authorization, nor expect they will put up with having to enter a password over and over.
OpenSSH Session Multiplexing
    This solution works even when using earlier versions of OpenSSH where the
    ControlPersistoption isn't available. (Working bash example at end of this answer)
Note: OpenSSH 3.9 introduced Session Multiplexing over a "control master connection" (in 2005), However, the ControlPersist option wasn't introduced until OpenSSH 5.6 (released in 2010).
ssh session multiplexing allows a script to authenticate once and do multiple ssh transactions over the authenticated connection. For example, if you have a script that runs several distinct tasks using ssh, scp, or sftp, each transaction can be carried out over OpenSSH 'control master session' that refers to location of its named-socket in the filesystem.
The following one-time-password authentication is useful when running a script that has to perform multiple ssh operations and one wants to avoid users having to password authenticate more than once, and is especially useful in cases where public key authentication isn't viable - e.g. not permitted, or at least not configured.
Most solutions I've seen entail using ControlPersist to tell ssh to keep the control master connection open, either indefinitely, or for some specific number of seconds.
Unfortunately, systems with OpenSSH prior to 5.6 don't have that option (wherein upgrading them might not be feasible). Unfortunately, there doesn't seem to be much documentation or discussion about that limitation online.
Reading through old release docs I discovered ControlPersist arrived late in the game for ssh session multiplexing scene. implying there may have been an alternative way to configure session multiplexing without relying on the ControlPersist option prior to it.
Initially trying to configure persistent-sessions from command line options rather than the config parameter, I ran into the problem of the ssh session terminating prematurely, closing control connection client sessions with it, or, alternatively, the connection was held open (kept ssh control master alive), terminal I/O was blocked, and the script would hang.
The following clarifies how to accomplish it.
OpenSSH option ssh flag Purpose
------------------- --------- -----------------------------
-o ControlMaster=yes -M Establishes sharable connection
-o ControlPath=path -S path Specifies path of connection's named socket
-o ControlPersist=600 Keep shareable connection open 10 min.
-o ControlPersist=yes Keep shareable connection open indefinitely
-N Don't create shell or run a command
-f Go into background after authenticating
-O exit Closes persistent connection
ControlPersist form Equivalent Purpose
------------------- ---------------- -------------------------
-o ControlPersist=yes ssh -Nf Keep control connection open indefinitely
-o ControlPersist=300 ssh -f sleep 300 Keep control connection open 5 min.
Note: scp and sftp implement -S flag differently, and -M flag not at all, so, for those commands, the -o option form is always required.
Sketchy Overview of Operations:
Note: This incomplete example doesn't execute as shown.
ctl=<path to dir to store named socket>
ssh -fNMS $ctl user#host # open control master connection
ssh -S $ctl … # example of ssh over connection
scp -o ControlPath=$ctl … # example of scp over connection
sftp -o ControlPath=$ctl … # example of sftp over connection
ssh -S $ctl -O exit # close control master connection
Session Multiplexing Demo
(Try it. You'll like it. Working example - authenticates only once):
Running this script will probably help you understand it quicker than reading it, and it is fascinating.
Note: If you lack access to remote host, just enter localhost at the "Host...?" prompt if you want to try this demo script
#!/bin/bash # This script demonstrates ssh session multiplexing
trap "[ -z "$ctl" ] || ssh -S $ctl -O exit $user#$host" EXIT # closes conn, deletes fifo
read -p "Host to connect to? " host
read -p "User to login with? " user
BOLD="\n$(tput bold)"; NORMAL="$(tput sgr0)"
echo -e "${BOLD}Create authenticated persistent control master connection:${NORMAL}"
sshfifos=~/.ssh/controlmasters
[ -d $sshfifos ] || mkdir -p $sshfifos; chmod 755 $sshfifos
ctl=$sshfifos/$user#$host:22 # ssh stores named socket ctrl conn here
ssh -fNMS $ctl $user#$host # Control Master: Prompts passwd then persists in background
lcldir=$(mktemp -d /tmp/XXXX)
echo -e "\nLocal dir: $lcldir"
rmtdir=$(ssh -S $ctl $user#$host "mktemp -d /tmp/XXXX")
echo "Remote dir: $rmtdir"
echo -e "${BOLD}Copy self to remote with scp:${NORMAL}"
scp -o ControlPath=$ctl ${BASH_SOURCE[0]} $user#$host:$rmtdir
echo -e "${BOLD}Display 4 lines of remote script, with ssh:${NORMAL}"
echo "====================================================================="
echo $rmtdir | ssh -S $ctl $user#$host "dir=$(</dev/stdin); head -4 \$dir/*"
echo "====================================================================="
echo -e "${BOLD}Do some pointless things with sftp:${NORMAL}"
sftp -o ControlPath=$ctl $user#$host:$rmtdir <<EOF
pwd
ls
lcd $lcldir
get *
quit
EOF
Using a master control socket, you can use multiple processes without having to authenticate more than once. This is just a simple example; see man ssh_config under ControlPath for advice on using a more secure socket.
It's not quite clear what you mean by sourcing somecommand locally; I'm going to assume it is a local script that you want copied over to the remote host. The simplest thing to do is just copy it over to run it.
# Copy the first file, and tell ssh to keep the connection open
# in the background after scp completes
$ scp -o ControlMaster=yes -o ControlPersist=yes -o ControlPath=%C somefile user#host:/tmp/somefile
# Copy the script on the same connection
$ scp -o ControlPath=%C somecommand user#host:
# Run the script on the same connection
$ ssh -o ControlPath=%C user#host somecommand
# Close the connection
$ ssh -o ControlPath=%C -O exit user#host
Of course, the user could use public key authentication to avoid entering their credentials at all, but ssh would still go through the authentication process each time. Here, the authentication process is only done once, by the command using ControlMaster=yes. The other two processes reuse that connection. The last commmand, with -O exit, doesn't actually connect; it just tells the local connection to close itself.
$ echo "abc" | cat <(echo "def")
The expression <(echo "def") expands to a file name, typically something like /dev/fd/63, that names a (virtual) file containing the text "def". So lets's simplify it a bit:
$ echo "def" > def.txt
$ echo "abc" | cat def.txt
This will also prints just def.
The pipe does feed the line abc to the standard input of the cat command. But because cat is given a file name on its command line, it doesn't read from its standard input. The abc is just quietly ignored, and the cat command prints the contents of the named file -- which is exactly what you told it to do.
The problem with echo abc | cat <(echo def) is that the <() wins the "providing the input" race. Luckily, bash will allow you to supply many inputs using mulitple <() constructs. So the trick is, how do you get the output of your echo abc into the <()?
How about:
$ echo abc | cat <(echo def) <(cat)
def
abc
If you need to handle the input from the pipe first, just switch the order:
$ echo abc | cat <(cat) <(echo def)
abc
def

Terminating SSH session executed by bash script

I have a script I can run locally to remotely start a server:
#!/bin/bash
ssh user#host.com <<EOF
nohup /path/to/run.sh &
EOF
echo 'done'
After running nohup, it hangs. I have to hit ctrl-c to exit the script.
I've tried adding an explicit exit at the end of the here doc and using "-t" argument for ssh. Neither works. How do I make this script exit immediately?
EDIT: The client is OSX 10.6, server is Ubuntu.
I think the problem is that nohup can't redirect output when you come in from ssh, it only redirects to nohup.out when it thinks it's connected to a terminal, and I the stdin override you have will prevent that, even with -t.
A workaround might be to redirect the output yourself, then the ssh client can disconnect - it's not waiting for the stream to close. Something like:
nohup /path/to/run.sh > run.log &
(This worked for me in a simple test connecting to an Ubuntu server from an OS X client.)
The problem might be that ...
... ssh is respecting the POSIX standard when not closing the session
if a process is still attached to the tty.
Therefore a solution might be to detach the stdin of the nohup command from the tty:
nohup /path/to/run.sh </dev/null &
See: SSH Hangs On Exit When Using nohup
Yet another approach might be to use ssh -t -t to force pseudo-tty allocation even if stdin isn't a terminal.
man ssh | less -Ip 'multiple -t'
ssh -t -t user#host.com <<EOF
nohup /path/to/run.sh &
EOF
See: BASH spawn subshell for SSH and continue with program flow
Redirecting the stdin of the remote host from a here document while invoking ssh without an explicit command leads to the message: Pseudo-terminal will not be allocated because stdin is not a terminal.
To avoid this message either use ssh's -T switch to tell the remote host there is no need to allocate a pseudo-terminal or explicitly specify a command (such as /bin/sh) for the remote host to execute the commands provided by the here document.
If an explicit command is given to ssh, the default is to provide no login shell in the form of a pseudo-terminal, i. e. there will be no normal login session when a command is specified (see man ssh).
Without a command specified for ssh, on the other hand, the default is to create a pseudo-tty for an interactive login session on the remote host.
- ssh user#host.com <<EOF
+ ssh -T user#host.com <<EOF
+ ssh user#host.com /bin/bash <<EOF
As a rule, ssh -t or even ssh -t -t should only be used if there are commands that expect stdin / stdout to be a terminal (such as top or vim) or if it is necessary to kill the remote shell and its children when the ssh client command finishes execution (see: ssh command unexpectedly continues on other system after ssh terminates).
As far as I can tell, the only way to combine an ssh command that does not allocate a pseudo-tty and a nohup command that writes to nohup.out on the remote host is to let the nohup command execute in a pseudo-terminal not created by the ssh mechanism. This can be done with the script command, for example, and will avoid the tcgetattr: Inappropriate ioctl for device message.
#!/bin/bash
ssh localhost /bin/sh <<EOF
#0<&- script -q /dev/null nohup sleep 10 1>&- &
#0<&- script -q -c "nohup sh -c 'date; sleep 10 1>&- &'" /dev/null # Linux
0<&- script -q /dev/null nohup sh -c 'date; sleep 10 1>&- &' # FreeBSD, Mac OS X
cat nohup.out
exit 0
EOF
echo 'done'
exit 0
You need to add a exit 0 at the end.

How do I kill a backgrounded/detached ssh session?

I am using the program synergy together with an ssh tunnel
It works, i just have to open an console an type these two commands:
ssh -f -N -L localhost:12345:otherHost:12345 otherUser#OtherHost
synergyc localhost
because im lazy i made an Bash-Script which is run with one mouseclick on an icon:
#!/bin/bash
ssh -f -N -L localhost:12345:otherHost:12345 otherUser#OtherHost
synergyc localhost
the Bash-Script above works as well, but now i also want to kill synergy and the ssh tunnel via one mouseclick, so i have to save the PIDs of synergy and ssh into file to kill them later:
#!/bin/bash
mkdir -p /tmp/synergyPIDs || exit 1
rm -f /tmp/synergyPIDs/ssh || exit 1
rm -f /tmp/synergyPIDs/synergy || exit 1
[ ! -e /tmp/synergyPIDs/ssh ] || exit 1
[ ! -e /tmp/synergyPIDs/synergy ] || exit 1
ssh -f -N -L localhost:12345:otherHost:12345 otherUser#OtherHost
echo $! > /tmp/synergyPIDs/ssh
synergyc localhost
echo $! > /tmp/synergyPIDs/synergy
But the files of this script are empty.
How do I get the PIDs of ssh and synergy?
(I try to avoid ps aux | grep ... | awk ... | sed ... combinations, there has to be an easier way.)
With all due respect to the users of pgrep, pkill, ps | awk, etc, there is a much better way.
Consider that if you rely on ps -aux | grep ... to find a process you run the risk of a collision. You may have a use case where that is unlikely, but as a general rule, it's not the way to go.
SSH provides a mechanism for managing and controlling background processes. But like so many SSH things, it's an "advanced" feature, and many people (it seems, from the other answers here) are unaware of its existence.
In my own use case, I have a workstation at home on which I want to leave a tunnel that connects to an HTTP proxy on the internal network at my office, and another one that gives me quick access to management interfaces on co-located servers. This is how you might create the basic tunnels, initiated from home:
$ ssh -fNT -L8888:proxyhost:8888 -R22222:localhost:22 officefirewall
$ ssh -fNT -L4431:www1:443 -L4432:www2:443 colocatedserver
These cause ssh to background itself, leaving the tunnels open. But if the tunnel goes away, I'm stuck, and if I want to find it, I have to parse my process list and home I've got the "right" ssh (in case I've accidentally launched multiple ones that look similar).
Instead, if I want to manage multiple connections, I use SSH's ControlMaster config option, along with the -O command-line option for control. For example, with the following in my ~/.ssh/config file,
host officefirewall colocatedserver
ControlMaster auto
ControlPath ~/.ssh/cm_sockets/%r#%h:%p
the ssh commands above, when run, will leave spoor in ~/.ssh/cm_sockets/ which can then provide access for control, for example:
$ ssh -O check officefirewall
Master running (pid=23980)
$ ssh -O exit officefirewall
Exit request sent.
$ ssh -O check officefirewall
Control socket connect(/home/ghoti/.ssh/cm_socket/ghoti#192.0.2.5:22): No such file or directory
And at this point, the tunnel (and controlling SSH session) is gone, without the need to use a hammer (kill, killall, pkill, etc).
Bringing this back to your use-case...
You're establishing the tunnel through which you want syngergyc to talk to syngergys on TCP port 12345. For that, I'd do something like the following.
Add an entry to your ~/.ssh/config file:
Host otherHosttunnel
HostName otherHost
User otherUser
LocalForward 12345 otherHost:12345
RequestTTY no
ExitOnForwardFailure yes
ControlMaster auto
ControlPath ~/.ssh/cm_sockets/%r#%h:%p
Note that the command line -L option is handled with the LocalForward keyword, and the Control{Master,Path} lines are included to make sure you have control after the tunnel is established.
Then, you might modify your bash script to something like this:
#!/bin/bash
if ! ssh -f -N otherHosttunnel; then
echo "ERROR: couldn't start tunnel." >&2
exit 1
else
synergyc localhost
ssh -O exit otherHosttunnel
fi
The -f option backgrounds the tunnel, leaving a socket on your ControlPath to close the tunnel later. If the ssh fails (which it might due to a network error or ExitOnForwardFailure), there's no need to exit the tunnel, but if it did not fail (else), synergyc is launched and then the tunnel is closed after it exits.
You might also want to look in to whether the SSH option LocalCommand could be used to launch synergyc from right within your ssh config file.
Quick summary: Will not work.
My first idea is that you need to start the processes in the background to get their PIDs with $!.
A pattern like
some_program &
some_pid=$!
wait $some_pid
might do what you need... except that then ssh won't be in the foreground to ask for passphrases any more.
Well then, you might need something different after all. ssh -f probably spawns a new process your shell can never know from invoking it anyway. Ideally, ssh itself would offer a way to write its PID into some file.
just came across this thread and wanted to mention the "pidof" linux utility:
$ pidof init
1
You can use lsof to show the pid of the process listening to port 12345 on localhost:
lsof -t -i #localhost:12345 -sTCP:listen
Examples:
PID=$(lsof -t -i #localhost:12345 -sTCP:listen)
lsof -t -i #localhost:12345 -sTCP:listen >/dev/null && echo "Port in use"
well i dont want to add an & at the end of the commands as the connection will die if the console wintow is closed ... so i ended up with an ps-grep-awk-sed-combo
ssh -f -N -L localhost:12345:otherHost:12345 otherUser#otherHost
echo `ps aux | grep -F 'ssh -f -N -L localhost' | grep -v -F 'grep' | awk '{ print $2 }'` > /tmp/synergyPIDs/ssh
synergyc localhost
echo `ps aux | grep -F 'synergyc localhost' | grep -v -F 'grep' | awk '{ print $2 }'` > /tmp/synergyPIDs/synergy
(you could integrate grep into awk, but im too lazy now)
You can drop the -f, which makes it run it in background, then run it with eval and force it to the background yourself.
You can then grab the pid. Make sure to put the & within the eval statement.
eval "ssh -N -L localhost:12345:otherHost:12345 otherUser#OtherHost & "
tunnelpid=$!
Another option is to use pgrep to find the PID of the newest ssh process
ssh -fNTL 8073:localhost:873 otherUser#OtherHost
tunnelPID=$(pgrep -n -x ssh)
synergyc localhost
kill -HUP $tunnelPID
This is more of a special case for synergyc (and most other programs that try to daemonize themselves). Using $! would work, except that synergyc does a clone() syscall during execution that will give it a new PID other than the one that bash thought it has. If you want to get around this so that you can use $!, then you can tell synergyc to stay in the forground and then background it.
synergyc -f -n mydesktop remoteip &
synergypid=$!
synergyc also does a few other things like autorestart that you may want to turn off if you are trying to manage it.
Based on the very good answer of #ghoti, here is a simpler script (for testing) utilising the SSH control sockets without the need of extra configuration:
#!/bin/bash
if ssh -fN -MS /tmp/mysocket -L localhost:12345:otherHost:12345 otherUser#otherHost; then
synergyc localhost
ssh -S /tmp/mysocket -O exit otherHost
fi
synergyc will be only started if tunnel has been established successfully, which itself will be closed as soon as synergyc returns.
Albeit the solution lacks proper error reporting.
You could look out for the ssh proceess that is bound to your local port, using this line:
netstat -tpln | grep 127\.0\.0\.1:12345 | awk '{print $7}' | sed 's#/.*##'
It returns the PID of the process using port 12345/TCP on localhost. So you don't have to filter all ssh results from ps.
If you just need to check, if that port is bound, use:
netstat -tln | grep 127\.0\.0\.1:12345 >/dev/null 2>&1
Returns 1 if none bound or 0 if someone is listening to this port.
There are many interesting answers here, but nobody mentioned that the manpage of SSH does describe this exact case! (see TCP FORWARDING section). And the solution they offer is much simpler:
ssh -fL 12345:localhost:12345 user#remoteserver sleep 10
synergyc localhost
Now in details:
First we start SSH with a tunnel; thanks to -f it will initiate the connection and only then fork to background (unlike solutions with ssh ... &; pid=$! where ssh is sent to background and next command is executed before the tunnel is created). On the remote machine it will run sleep 10 which will wait 10 seconds and then end.
Within 10 seconds, we should start our desired command, in this case synergyc localhost. It will connect to the tunnel and SSH will then know that the tunnel is in use.
After 10 seconds pass, sleep 10 command will finish. But the tunnel is still in use by synergyc, so SSH will not close the underlying connection until the tunnel is released (i.e. until synergyc closes socket).
When synergyc is closed, it will release the tunnel, and SSH in turn will terminate itself, closing a connection.
The only downside of this approach is that if the program we use will close and re-open connection for some reason then SSH will close the tunnel right after connection is closed, and the program won't be able to reconnect. If this is an issue then you should use an approach described in #doak's answer which uses control socket to properly terminate SSH connection and uses -f to make sure tunnel is created when SSH forks to the background.

Getting ssh to execute a command in the background on target machine

This is a follow-on question to the How do you use ssh in a shell script? question. If I want to execute a command on the remote machine that runs in the background on that machine, how do I get the ssh command to return? When I try to just include the ampersand (&) at the end of the command it just hangs. The exact form of the command looks like this:
ssh user#target "cd /some/directory; program-to-execute &"
Any ideas? One thing to note is that logins to the target machine always produce a text banner and I have SSH keys set up so no password is required.
I had this problem in a program I wrote a year ago -- turns out the answer is rather complicated. You'll need to use nohup as well as output redirection, as explained in the wikipedia artcle on nohup, copied here for your convenience.
Nohuping backgrounded jobs is for
example useful when logged in via SSH,
since backgrounded jobs can cause the
shell to hang on logout due to a race
condition [2]. This problem can also
be overcome by redirecting all three
I/O streams:
nohup myprogram > foo.out 2> foo.err < /dev/null &
This has been the cleanest way to do it for me:-
ssh -n -f user#host "sh -c 'cd /whereever; nohup ./whatever > /dev/null 2>&1 &'"
The only thing running after this is the actual command on the remote machine
Redirect fd's
Output needs to be redirected with &>/dev/null which redirects both stderr and stdout to /dev/null and is a synonym of >/dev/null 2>/dev/null or >/dev/null 2>&1.
Parantheses
The best way is to use sh -c '( ( command ) & )' where command is anything.
ssh askapache 'sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'
Nohup Shell
You can also use nohup directly to launch the shell:
ssh askapache 'nohup sh -c "( ( chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'
Nice Launch
Another trick is to use nice to launch the command/shell:
ssh askapache 'nice -n 19 sh -c "( ( nohup chown -R ask:ask /www/askapache.com &>/dev/null ) & )"'
If you don't/can't keep the connection open you could use screen, if you have the rights to install it.
user#localhost $ screen -t remote-command
user#localhost $ ssh user#target # now inside of a screen session
user#remotehost $ cd /some/directory; program-to-execute &
To detach the screen session: ctrl-a d
To list screen sessions:
screen -ls
To reattach a session:
screen -d -r remote-command
Note that screen can also create multiple shells within each session. A similar effect can be achieved with tmux.
user#localhost $ tmux
user#localhost $ ssh user#target # now inside of a tmux session
user#remotehost $ cd /some/directory; program-to-execute &
To detach the tmux session: ctrl-b d
To list screen sessions:
tmux list-sessions
To reattach a session:
tmux attach <session number>
The default tmux control key, 'ctrl-b', is somewhat difficult to use but there are several example tmux configs that ship with tmux that you can try.
I just wanted to show a working example that you can cut and paste:
ssh REMOTE "sh -c \"(nohup sleep 30; touch nohup-exit) > /dev/null &\""
You can do this without nohup:
ssh user#host 'myprogram >out.log 2>err.log &'
Quickest and easiest way is to use the 'at' command:
ssh user#target "at now -f /home/foo.sh"
I think you'll have to combine a couple of these answers to get what you want. If you use nohup in conjunction with the semicolon, and wrap the whole thing in quotes, then you get:
ssh user#target "cd /some/directory; nohup myprogram > foo.out 2> foo.err < /dev/null"
which seems to work for me. With nohup, you don't need to append the & to the command to be run. Also, if you don't need to read any of the output of the command, you can use
ssh user#target "cd /some/directory; nohup myprogram > /dev/null 2>&1"
to redirect all output to /dev/null.
This worked for me may times:
ssh -x remoteServer "cd yourRemoteDir; ./yourRemoteScript.sh </dev/null >/dev/null 2>&1 & "
You can do it like this...
sudo /home/script.sh -opt1 > /tmp/script.out &
It appeared quite convenient for me to have a remote tmux session using the tmux new -d <shell cmd> syntax like this:
ssh someone#elsewhere 'tmux new -d sleep 600'
This will launch new session on elsewhere host and ssh command on local machine will return to shell almost instantly. You can then ssh to the remote host and tmux attach to that session. Note that there's nothing about local tmux running, only remote!
Also, if you want your session to persist after the job is done, simply add a shell launcher after your command, but don't forget to enclose in quotes:
ssh someone#elsewhere 'tmux new -d "~/myscript.sh; bash"'
Actually, whenever I need to run a command on a remote machine that's complicated, I like to put the command in a script on the destination machine, and just run that script using ssh.
For example:
# simple_script.sh (located on remote server)
#!/bin/bash
cat /var/log/messages | grep <some value> | awk -F " " '{print $8}'
And then I just run this command on the source machine:
ssh user#ip "/path/to/simple_script.sh"
If you run remote command without allocating tty, redirect stdout/stderr works, nohup is not necessary.
ssh user#host 'background command &>/dev/null &'
If you use -t to allocate tty to run interactive command along with background command, and background command is the last command, like this:
ssh -t user#host 'bash -c "interactive command; nohup backgroud command &>/dev/null &"'
It's possible that background command doesn't actually start. There's race here:
bash exits after nohup starts. As a session leader, bash exit results in HUP signal sent to nohup process.
nohup ignores HUP signal.
If 1 completes before 2, the nohup process will exit and won't start the background command at all. We need to wait nohup start the background command. A simple workaroung is to just add a sleep:
ssh -t user#host 'bash -c "interactive command; nohup backgroud command &>/dev/null & sleep 1"'
The question was asked and answered years ago, I don't know if openssh behavior changed since then. I was testing on:
OpenSSH_8.6p1, OpenSSL 1.1.1g FIPS 21 Apr 2020
I was trying to do the same thing, but with the added complexity that I was trying to do it from Java. So on one machine running java, I was trying to run a script on another machine, in the background (with nohup).
From the command line, here is what worked: (you may not need the "-i keyFile" if you don't need it to ssh to the host)
ssh -i keyFile user#host bash -c "\"nohup ./script arg1 arg2 > output.txt 2>&1 &\""
Note that to my command line, there is one argument after the "-c", which is all in quotes. But for it to work on the other end, it still needs the quotes, so I had to put escaped quotes within it.
From java, here is what worked:
ProcessBuilder b = new ProcessBuilder("ssh", "-i", "keyFile", "bash", "-c",
"\"nohup ./script arg1 arg2 > output.txt 2>&1 &\"");
Process process = b.start();
// then read from process.getInputStream() and close it.
It took a bit of trial & error to get this working, but it seems to work well now.
YOUR-COMMAND &> YOUR-LOG.log &
This should run the command and assign a process id you can simply tail -f YOUR-LOG.log to see results written to it as they happen. you can log out anytime and the process will carry on
If you are using zsh then use program-to-execute &! is a zsh-specific shortcut to both background and disown the process, such that exiting the shell will leave it running.
A follow-on to #cmcginty's concise working example which also shows how to alternatively wrap the outer command in double quotes. This is how the template would look if invoked from within a PowerShell script (which can only interpolate variables from within double-quotes and ignores any variable expansion when wrapped in single quotes):
ssh user#server "sh -c `"($cmd) &>/dev/null </dev/null &`""
Inner double-quotes are escaped with back-tick instead of backslash. This allows $cmd to be composed by the PowerShell script, e.g. for deployment scripts and automation and the like. $cmd can even contain a multi-line heredoc if composed with unix LF.
First follow this procedure:
Log in on A as user a and generate a pair of authentication keys. Do not enter a passphrase:
a#A:~> ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/a/.ssh/id_rsa):
Created directory '/home/a/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/a/.ssh/id_rsa.
Your public key has been saved in /home/a/.ssh/id_rsa.pub.
The key fingerprint is:
3e:4f:05:79:3a:9f:96:7c:3b:ad:e9:58:37:bc:37:e4 a#A
Now use ssh to create a directory ~/.ssh as user b on B. (The directory may already exist, which is fine):
a#A:~> ssh b#B mkdir -p .ssh
b#B's password:
Finally append a's new public key to b#B:.ssh/authorized_keys and enter b's password one last time:
a#A:~> cat .ssh/id_rsa.pub | ssh b#B 'cat >> .ssh/authorized_keys'
b#B's password:
From now on you can log into B as b from A as a without password:
a#A:~> ssh b#B
then this will work without entering a password
ssh b#B "cd /some/directory; program-to-execute &"
I think this is what you need:
At first you need to install sshpass on your machine.
then you can write your own script:
while read pass port user ip; do
sshpass -p$pass ssh -p $port $user#$ip <<ENDSSH1
COMMAND 1
.
.
.
COMMAND n
ENDSSH1
done <<____HERE
PASS PORT USER IP
. . . .
. . . .
. . . .
PASS PORT USER IP
____HERE

Resources