I have a perl script that pulls hostnames from a database and ssh's to them en masse with bash scripts piped to those ssh sessions for monitoring purposes.
These hosts overwhelmingly have root keys and in addition to monitoring I'm trying to justify further use of root keys to better facilitate this monitoring method.
Here is an example (the context is a while loop in the perl script):
system("cat /usr/local/bin/filesystem_scan.sh | ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root\#$name '/bin/bash; uname -a &> /dev/null' ");
There is another bash script that runs the perl one and gets result codes from the script that was piped and edits them into a report.
#!/bin/bash
perl /usr/local/bin/readonly-fs-scan.pl | grep -v " 0$\| 127$" | column -t > report.txt
The problem is that the script takes forever to run (over 500 hosts) because of approximately 50 hosts that don't have root keys. As a stop-gap to getting approval for keys on those servers I'm trying to find a way to implement expect into this process so that return is sent any time a password is requested (otherwise the script stalls for over a minute waiting for input for each of those 50 hosts, which causes a script that should take about 5 minutes to run to take about an hour). I've been able to do this in a simple bash script that ssh's to one server like so:
#!/bin/bash
expect -c "set timeout 5
spawn ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root#$name
expect \"*password:\"
send \"\r\""
However I'm struggling to grasp how I can insert this into the perl script.
This is a work assignment so I'm limited in large-scope changes I can make - (changing code language, installing extra utilities, etc.)
Related
I am passing a file from one server to another using authorized keys.
However, in the event keys are no longer valid, the script is being asked for the password on and on.
I have tried
scp ${user}#$host}:/tmp ./file1 </dev/null
but I still get the prompt.
The only time when this works is if I run it off schedule like this:
echo "scp ${user}#$host}:/tmp ./file1" | at now
In this case it will correctly error out if keys are no longer valid.
But how can I create a blank input stream, that will not be prompting the user if the script is run interactively?
#David:
echo "" | scp ${user}#${host}:/tmp ./file1 </dev/null
didn't help, same response, so it may need to have an stty command to zero tty input, I'm guessing now, per Kenster's note.
But how can I create a blank input stream, that will not be prompting the user if the script is run interactively?
Instead of that disable password authentication.
scp -o BatchMode=yes -o PasswordAuthentication=no -o PubkeyAuthentication=yes ...
This might work as well
scp ${user}#${host}:/tmp ./file1 /dev/tty0 > /dev/null
I have 4 shell commands I need to run and they do not depend on each other.
I have 4 slave machines. So, I want to run one of the 4 commands on each of the 4 machines, and then I want to wait until all 4 of them are finished.
How do I distribute this processing? This is what I tried:
$1 is a list of ip addresses to the slave machines.
for host in $(cat $1)
do
echo $host
# ssh into each machine and launch command
ssh username#$host <command>;
done
But this seems as if it is waiting for the command to finish before moving on to the next host and launching the next command.
How do I accomplish this distributed processing that doesn't depend on each other?
I would use GNU Parallel like this - running hostname in parallel on each of 4 servers:
parallel -j 4 --nonall -S 192.168.0.1,192.168.0.2,192.168.0.3,192.168.0.4 hostname
If you need to pass parameters, use --onall and put arguments after :::
parallel -j 4 --onall -S 192.168.0.1,192.168.0.2,192.168.0.3,192.168.0.4 echo ::: hello
Add --tag if you want the output lines tagged by the hostname/IP.
Add -k if you want to keep the output in order.
Add : to the server list to run on local host too.
If you aren't concerned about how many commands run concurrently, just put each one in the background with &, then wait on them as a group.
while IFS= read -r host; do
ssh username#$host <command> &
done < "$1"
wait
Note the use of a while loop instead of a for loop; see Bash FAQ 001.
The ssh part of your script needs to be like:
$ ssh -f user#host "sh -c 'sleep 30 ; nohup ls > foo 2>&1 &'"
This one sleeps for 30 secs and writes the output of ls to file foo. 30 secs is enough for you to go and see it yourself. Just build your loop around that.
I got this trick from Solaris documentation, for copying ssh public keys to remote hosts.
note: ssh-copy-id isn't available on Solaris
$ cat some_data_file | ssh user#host "cat >/tmp/some_data_file; some_shell_cmd"
I wanted to adapt it to do more involved things.
Specifically I wanted some_shell_command to be a script sent from the local host to execute on the remote host... a script would interact with the local keyboard (e.g. prompt user when the script was running on the remote host).
I experimented with ways of sending multiple things over stdin from multiple sources. But certain things that work in in local shell don't work over ssh, and some things, such as the following, didn't do what I wanted at all:
$ echo "abc" | cat <(echo "def") # echoes: def (I wanted abc\ndef)
$ echo "abc" | cat < <(echo "def") # echoes: def (I wanted abc\ndef)
$ echo "abc" | cat <<-EOF
> echo $(</dev/stdin) #echoes: echo abc (I wanted: abc)
> EOF
# messed with eval for the above but that was a problem too.
#chepner concluded it's not feasible to do all of that in a single ssh command. He suggested a theoretical alternative that didn't work as hoped, but I got it working after some research and tweaking and documented the results of that and posted it as an answer to this question.
Without that solution, having to run multiple ssh, and scp commands by default entails being prompted for password multiple times, which is a major drag.
I can't expect all the users of a script I write in a multi-user environment to configure public key authorization, nor expect they will put up with having to enter a password over and over.
OpenSSH Session Multiplexing
This solution works even when using earlier versions of OpenSSH where the
ControlPersistoption isn't available. (Working bash example at end of this answer)
Note: OpenSSH 3.9 introduced Session Multiplexing over a "control master connection" (in 2005), However, the ControlPersist option wasn't introduced until OpenSSH 5.6 (released in 2010).
ssh session multiplexing allows a script to authenticate once and do multiple ssh transactions over the authenticated connection. For example, if you have a script that runs several distinct tasks using ssh, scp, or sftp, each transaction can be carried out over OpenSSH 'control master session' that refers to location of its named-socket in the filesystem.
The following one-time-password authentication is useful when running a script that has to perform multiple ssh operations and one wants to avoid users having to password authenticate more than once, and is especially useful in cases where public key authentication isn't viable - e.g. not permitted, or at least not configured.
Most solutions I've seen entail using ControlPersist to tell ssh to keep the control master connection open, either indefinitely, or for some specific number of seconds.
Unfortunately, systems with OpenSSH prior to 5.6 don't have that option (wherein upgrading them might not be feasible). Unfortunately, there doesn't seem to be much documentation or discussion about that limitation online.
Reading through old release docs I discovered ControlPersist arrived late in the game for ssh session multiplexing scene. implying there may have been an alternative way to configure session multiplexing without relying on the ControlPersist option prior to it.
Initially trying to configure persistent-sessions from command line options rather than the config parameter, I ran into the problem of the ssh session terminating prematurely, closing control connection client sessions with it, or, alternatively, the connection was held open (kept ssh control master alive), terminal I/O was blocked, and the script would hang.
The following clarifies how to accomplish it.
OpenSSH option ssh flag Purpose
------------------- --------- -----------------------------
-o ControlMaster=yes -M Establishes sharable connection
-o ControlPath=path -S path Specifies path of connection's named socket
-o ControlPersist=600 Keep shareable connection open 10 min.
-o ControlPersist=yes Keep shareable connection open indefinitely
-N Don't create shell or run a command
-f Go into background after authenticating
-O exit Closes persistent connection
ControlPersist form Equivalent Purpose
------------------- ---------------- -------------------------
-o ControlPersist=yes ssh -Nf Keep control connection open indefinitely
-o ControlPersist=300 ssh -f sleep 300 Keep control connection open 5 min.
Note: scp and sftp implement -S flag differently, and -M flag not at all, so, for those commands, the -o option form is always required.
Sketchy Overview of Operations:
Note: This incomplete example doesn't execute as shown.
ctl=<path to dir to store named socket>
ssh -fNMS $ctl user#host # open control master connection
ssh -S $ctl … # example of ssh over connection
scp -o ControlPath=$ctl … # example of scp over connection
sftp -o ControlPath=$ctl … # example of sftp over connection
ssh -S $ctl -O exit # close control master connection
Session Multiplexing Demo
(Try it. You'll like it. Working example - authenticates only once):
Running this script will probably help you understand it quicker than reading it, and it is fascinating.
Note: If you lack access to remote host, just enter localhost at the "Host...?" prompt if you want to try this demo script
#!/bin/bash # This script demonstrates ssh session multiplexing
trap "[ -z "$ctl" ] || ssh -S $ctl -O exit $user#$host" EXIT # closes conn, deletes fifo
read -p "Host to connect to? " host
read -p "User to login with? " user
BOLD="\n$(tput bold)"; NORMAL="$(tput sgr0)"
echo -e "${BOLD}Create authenticated persistent control master connection:${NORMAL}"
sshfifos=~/.ssh/controlmasters
[ -d $sshfifos ] || mkdir -p $sshfifos; chmod 755 $sshfifos
ctl=$sshfifos/$user#$host:22 # ssh stores named socket ctrl conn here
ssh -fNMS $ctl $user#$host # Control Master: Prompts passwd then persists in background
lcldir=$(mktemp -d /tmp/XXXX)
echo -e "\nLocal dir: $lcldir"
rmtdir=$(ssh -S $ctl $user#$host "mktemp -d /tmp/XXXX")
echo "Remote dir: $rmtdir"
echo -e "${BOLD}Copy self to remote with scp:${NORMAL}"
scp -o ControlPath=$ctl ${BASH_SOURCE[0]} $user#$host:$rmtdir
echo -e "${BOLD}Display 4 lines of remote script, with ssh:${NORMAL}"
echo "====================================================================="
echo $rmtdir | ssh -S $ctl $user#$host "dir=$(</dev/stdin); head -4 \$dir/*"
echo "====================================================================="
echo -e "${BOLD}Do some pointless things with sftp:${NORMAL}"
sftp -o ControlPath=$ctl $user#$host:$rmtdir <<EOF
pwd
ls
lcd $lcldir
get *
quit
EOF
Using a master control socket, you can use multiple processes without having to authenticate more than once. This is just a simple example; see man ssh_config under ControlPath for advice on using a more secure socket.
It's not quite clear what you mean by sourcing somecommand locally; I'm going to assume it is a local script that you want copied over to the remote host. The simplest thing to do is just copy it over to run it.
# Copy the first file, and tell ssh to keep the connection open
# in the background after scp completes
$ scp -o ControlMaster=yes -o ControlPersist=yes -o ControlPath=%C somefile user#host:/tmp/somefile
# Copy the script on the same connection
$ scp -o ControlPath=%C somecommand user#host:
# Run the script on the same connection
$ ssh -o ControlPath=%C user#host somecommand
# Close the connection
$ ssh -o ControlPath=%C -O exit user#host
Of course, the user could use public key authentication to avoid entering their credentials at all, but ssh would still go through the authentication process each time. Here, the authentication process is only done once, by the command using ControlMaster=yes. The other two processes reuse that connection. The last commmand, with -O exit, doesn't actually connect; it just tells the local connection to close itself.
$ echo "abc" | cat <(echo "def")
The expression <(echo "def") expands to a file name, typically something like /dev/fd/63, that names a (virtual) file containing the text "def". So lets's simplify it a bit:
$ echo "def" > def.txt
$ echo "abc" | cat def.txt
This will also prints just def.
The pipe does feed the line abc to the standard input of the cat command. But because cat is given a file name on its command line, it doesn't read from its standard input. The abc is just quietly ignored, and the cat command prints the contents of the named file -- which is exactly what you told it to do.
The problem with echo abc | cat <(echo def) is that the <() wins the "providing the input" race. Luckily, bash will allow you to supply many inputs using mulitple <() constructs. So the trick is, how do you get the output of your echo abc into the <()?
How about:
$ echo abc | cat <(echo def) <(cat)
def
abc
If you need to handle the input from the pipe first, just switch the order:
$ echo abc | cat <(cat) <(echo def)
abc
def
i have a script that fires remote commands on several different machines through ssh connection. Script goes something like:
for server in list; do
echo "output from $server"
ssh to server execute some command
done
The problem with this is evidently the time, as it needs to establish ssh connection, fire command, wait for answer, print it. What i would like is to have script that would try to establish connections all at once and return echo "output from $server" and output of command as soon as it gets it, so not necessary in the list order.
I've been googling this for a while but didn't find an answer. I cannot cancel ssh session after command run as one thread suggested, because i need an output and i cannot use parallel gnu suggested in other threads. Also i cannot use any other tool, i cannot bring/install anything on this machine, only useable tool is GNU bash, version 4.1.2(1)-release.
Another question is how are ssh sessions like this limited? If i simply paste 5+ or so lines of "ssh connect, do some command" it actually doesn't do anything, or execute only on first from list. (it works if i paste 3-4 lines). Thank you
Have you tried this?
for server in list; do
ssh user#server "command" &
done
wait
echo finished
Update: Start subshells:
for server in list; do
(echo "output from $server"; ssh user#server "command"; echo End $server) &
done
wait
echo All subshells finished
There are several parallel SSH tools that can handle that for you:
http://code.google.com/p/pdsh/
http://sourceforge.net/projects/clusterssh/
http://code.google.com/p/sshpt/
http://code.google.com/p/parallel-ssh/
Also, you could be interested in configuration deployment solutions such as Chef, Puppet, Ansible, Fabric, etc (see this summary ).
A third option is to use a terminal broadcast such as pconsole
If you only can use GNU commands, you can write your script like this:
for server in $servers ; do
( { echo "output from $server" ; ssh user#$server "command" ; } | \
sed -e "s/^/$server:/" ) &
done
wait
and then sort the output to reconcile the lines.
I started with the shell hacks mentionned in this thread, then proceeded to something somewhat more robust : https://github.com/bearstech/pussh
It's my daily workhorse, and I basically run anything against 250 servers in 20 seconds (it's actually rate limited otherwise the connection rate kills my ssh-agent). I've been using this for years.
See for yourself from the man page (clone it and run 'man ./pussh.1') : https://github.com/bearstech/pussh/blob/master/pussh.1
Examples
Show all servers rootfs usage in descending order :
pussh -f servers df -h / |grep /dev |sort -rn -k5
Count the number of processors in a cluster :
pussh -f servers grep ^processor /proc/cpuinfo |wc -l
Show the processor models, sorted by occurence :
pussh -f servers sed -ne "s/^model name.*: //p" /proc/cpuinfo |sort |uniq -c
Fetch a list of installed package in one file per host :
pussh -f servers -o packages-for-%h dpkg --get-selections
Mass copy a file tree (broadcast) :
tar czf files.tar.gz ... && pussh -f servers -i files.tar.gz tar -xzC /to/dest
Mass copy several remote file trees (gather) :
pussh -f servers -o '|(mkdir -p %h && tar -xzC %h)' tar -czC /src/path .
Note that the pussh -u feature (upload and execute) was the main reason why I programmed this, no tools seemed to be able to do this. I still wonder if that's the case today.
You may like the parallel-ssh project with the pssh command:
pssh -h servers.txt -l user command
It will output one line per server when the command is successfully executed. With the -P option you can also see the output of the command.
So I have a problem similar to how to send ssh job to background.
I have a windows c# program automated to execute tcpdump on a remote linux os using http://sshnet.codeplex.com/. I'm trying to execute tcpdump on the remote linux and leave it running after I disconnect.
I've been doing a lot of debugging using plink, but cannot seem to achieve the desired result. I've tried:
plink root#10.5.1.1 bash -c "tcpdump -i eth0 -w test.cap"
but it holds the sshclient until I ctrl+C (not going to work for automated solution). I've also tried variations of:
plink root#10.5.1.1 bash -c "tcpdump -i eth0 -w test.cap &"
but either the command is not executed at all (test.cap does not exist) or is terminated immediately (test.cap contains 1 line). During testing, I've left a ping going, so the capture should have somthing...
The previously mentioned link solves the problem with screen, but the remote linux os is not configurable and does not have screen. Any suggestions are welcome.
In the latter case, your tcpdump process is probably being aborted when you disconnect. Try:
plink root#10.5.1.1 bash -c "nohup tcpdump -i eth0 -w test.cap &"
See the manpage for nohup. You may also want to consider redirecting stdout and stderr to a file or /dev/null to prevent nohup from writing output to a file:
plink root#10.5.1.1 bash -c "nohup tcpdump -i eth0 -w test.cap >/dev/null 2>&1 &"
I had a similar problem while starting a remote application. This pattern worked for me on Debian servers:
ssh root#server "nohup /usr/local/bin/app -c cfg &; exit"
addition: for another test the above didn't work, ie. the command didn't start on the remote server. Adding a command that returns successfully before the exit seems to work.
ssh root#server "nohup /usr/local/bin/otherapp &; w; exit"
I had a similar situation:
(on windows machine) i wanted to create a ms batch script to open an SSH connection to a raspberry pi and execute a local script in the background.
I found that combining both Raj's and fahd's answers did the trick for me:
my ms batch script:
plink -load "raspberry Pi" -t -m startCommand.txt
the content of startCommand.txt is as follows:
nohup /home/pi/myscript >/dev/null 2>&1 &
w
exit
The ">/dev/null 2>&1 " is important!
I found out (the hard way) that the RPi's SDcard kept getting full by an extremely large nohup.out file (and with a full SDcard, the RPi couldn't even login properly)
reasoning:
I used the -load to load a saved session in PuTTY (i do this because i am authenticating with public/private keys instead of passwords, but this should be the same as simply typing in the host)
then -t (as recommended by Raj)
then -m to load a list of commands in that file
without the parameter "-t" and without the "w" and "exit", my batch script would just run, not execute 'myscript' and close again.
I had the same issue. I had a scrip in which I had nohup tcpdump .... & . I could not use ssh to run it as it dies when the ssh finished. The solution I came up with was super simple. I just added sleep 5 to the end of my script and it works just fine. It seems tcpdump needs some seconds to go to background safely before you exit even with nohup.
I had the same problem, and I found that the "-t" option seems to be important to nohup. I found the nohup wasn't taking affect without the "-t" option.
ssh -t user#remote 'nohup tcpdump -i any -w /tmp/somefile &>/dev/null & sleep 2'
I think that I've nailed it, at least in IBM AIX
I'm using
ssh -tq user#host "/path/start-tcpdump.ksh"
(authentication is done by publick key).
I was having inconsistent results using simple "nohup tcpdump .... &", sometimes it worked, sometimes it did not, sometimes it even blocked and I had to disconnect the session.
So far, this is working ok, I can't really say WHY it is working, but it is...
This is my start-tcpip.ksh
#!/usr/bin/ksh
HOST=$(uname -n)
FILTER="port not 22"
(tcpdump -i en1 -w $HOST-en1.cap $FILTER >/dev/null 2>&1 ) &
sleep 2
(tcpdump -i en2 -w $HOST-en2.cap $FILTER >/dev/null 2>&1 ) &
sleep 2
exit 0