Execute a command on remote hosts via ssh from inside a bash script - bash

I wrote a bash script which is supposed to read usernames and IP addresses from a file and execute a command on them via ssh.
This is hosts.txt :
user1 192.168.56.232
user2 192.168.56.233
This is myScript.sh :
cmd="ls -l"
while read line
do
set $line
echo "HOST:" $1#$2
ssh $1#$2 $cmd
exitStatus=$?
echo "Exit Status: " $exitStatus
done < hosts.txt
The problem is that execution seems to stop after the first host is done. This is the output:
$ ./myScript.sh
HOST: user1#192.168.56.232
total 2748
drwxr-xr-x 2 user1 user1 4096 2011-11-15 20:01 Desktop
drwxr-xr-x 2 user1 user1 4096 2011-11-10 20:37 Documents
...
drwxr-xr-x 2 user1 user1 4096 2011-11-10 20:37 Videos
Exit Status: 0
$
Why does is behave like this, and how can i fix it?

In your script, the ssh job gets the same stdin as the read line, and in your case happens to eat up all the lines on the first invocation. So read line only gets to see
the very first line of the input.
Solution: Close stdin for ssh, or better redirect from /dev/null. (Some programs
don't like having stdin closed)
while read line
do
ssh server somecommand </dev/null # Redirect stdin from /dev/null
# for ssh command
# (Does not affect the other commands)
printf '%s\n' "$line"
done < hosts.txt
If you don't want to redirect from /dev/null for every single job inside the loop, you can also try one of these:
while read line
do
{
commands...
} </dev/null # Redirect stdin from /dev/null for all
# commands inside the braces
done < hosts.txt
# In the following, let's not override the original stdin. Open hosts.txt on fd3
# instead
while read line <&3 # execute read command with fd0 (stdin) backed up from fd3
do
commands... # inside, you still have the original stdin
# (maybe the terminal) from outside, which can be practical.
done 3< hosts.txt # make hosts.txt available as fd3 for all commands in the
# loop (so fd0 (stdin) will be unaffected)
# totally safe way: close fd3 for all inner commands at once
while read line <&3
do
{
commands...
} 3<&-
done 3< hosts.txt

The problem that you are having is that the SSH process is consuming all of the stdin, so read doesn't see any of the input after the first ssh command has ran. You can use the -n flag for SSH to prevent this from happening, or you can redirect /dev/null to the stdin of the ssh command.
See the following for more information:
http://mywiki.wooledge.org/BashFAQ/089

Make sure the ssh command does not read from the hosts.txt using ssh -n

I have a feeling your question is unnecessarily verbose..
Essentially you should be able to reproduce the problem with:
while read line
do
echo $line
done < hosts.txt
Which should work just fine.. Do you edit the right file? Are there special characters in it? Check it with a proper editor (eg: vim).

Related

how to run bash script interactively from url? [duplicate]

I have a simple Bash script that takes in inputs and prints a few lines out with that inputs
fortinetTest.sh
read -p "Enter SSC IP: $ip " ip && ip=${ip:-1.1.1.1}
printf "\n"
#check IP validation
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "SSC IP: $ip"
printf "\n"
else
echo "Enter a valid SSC IP address. Ex. 1.1.1.1"
exit
fi
I tried to upload them into my server, then try to run it via curl
I am not sure why the input prompt never kick in when I use cURL/wget.
Am I missing anything?
With the curl ... | bash form, bash's stdin is reading the script, so stdin is not available for the read command.
Try using a Process Substitution to invoke the remote script like a local file:
bash <( curl -s ... )
Your issue can be simply be reproduced by run the script like below
$ cat test.sh | bash
Enter a valid SSC IP address. Ex. 1.1.1.1
This is because the bash you launch with a pipe is not getting a TTY, when you do a read -p it is read from stdin which is content of the test.sh in this case. So the issue is not with curl. The issue is not reading from the tty
So the fix is to make sure you ready it from tty
read < /dev/tty -p "Enter SSC IP: $ip " ip && ip=${ip:-1.1.1.1}
printf "\n"
#check IP validation
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "SSC IP: $ip"
printf "\n"
else
echo "Enter a valid SSC IP address. Ex. 1.1.1.1"
exit
fi
Once you do that even curl will start working
vagrant#vagrant:/var/www/html$ curl -s localhost/test.sh | bash
Enter SSC IP: 2.2.2.2
SSC IP: 2.2.2.2
I personally prefer source <(curl -s localhost/test.sh) option. While it is similar to bash ..., the one significant difference is how processes handled.
bash will result in a new process being spun up, and that process will evoke commands from the script.
source on the other hand will use current process to evoke commands from the script.
In some cases that can play a key role. I admit that is not very often though.
To demonstrate do the following:
### Open Two Terminals
# In the first terminal run:
echo "sleep 5" > ./myTest.sh
bash ./myTest.sh
# Switch to the second terminal and run:
ps -efjh
## Repeat the same with _source_ command
# In the first terminal run:
source ./myTest.sh
# Switch to the second terminal and run:
ps -efjh
Results should look similar to this:
Before execution:
Running bash (main + two subprocesses):
Running source (main + one subprocess):
UPDATE:
Difference in use variable usage by bash and source:
source command will use your current environment. Meaning that upon execution all changes and variable declarations, made by the script, will be available in your prompt.
bash on the other hand will be running in as a different process; therefore, all variables will be discarded when process exits.
I think everyone will agree that there are benefits and drawbacks to each method. You just have to decide which one is better for your use case.
## Test for variables declared by the script:
echo "test_var3='Some Other Value'" > ./myTest3.sh
bash ./myTest3.sh
echo $test_var3
source ./myTest3.sh
echo $test_var3
## Test for usability of current environment variables:
test_var="Some Value" # Setting a variable
echo "echo $test_var" > myTest2.sh # Creating a test script
chmod +x ./myTest2.sh # Adding execute permission
## Executing:
. myTest2.sh
bash ./myTest2.sh
source ./myTest2.sh
./myTest2.sh
## All of the above results should print the variable.
I hope this helps.

Shell script can read file line by line but not perform actions for each line

I'm trying to run this command over multiple machines
sshpass -p 'nico' ssh -o 'StrictHostKeyChecking=no' nico#x.x.x.x "mkdir test"
The IPs are stored in the following .txt file
$ cat ips.txt
10.0.2.15
10.0.2.5
I created a bash script that reads this file line by line. If I run it with an echo:
#!/bin/bash
input="ips.txt"
while IFS= read -r line
do
echo "$line"
#sshpass -p 'nico' ssh -o 'StrictHostKeyChecking=no' nico#$line "mkdir test"
done < "$input"
It prints every line:
$ ./variables.sh
10.0.2.15
10.0.2.5
This makes me understand that the script is working as intended. However, when I replace the echo line with the command I want to run for each line:
#!/bin/bash
input="ips.txt"
while IFS= read -r line
do
#echo "$line"
sshpass -p 'nico' ssh -o 'StrictHostKeyChecking=no' nico#$line "mkdir test"
done < "$input"
It only performs the action for the first IP on the file, then stops. Why?
Managed to solve this by using a for instead of a while. Script ended up looking like this:
for file in $(cat ips.txt)
do
sshpass -p 'nico' ssh -o 'StrictHostKeyChecking=no' nico#$file "mkdir test"
done
While your example is a solution that works, it's not the explanation.
Your could find the explanation here : ssh breaks out of while-loop in bash
In two words :
"while" loop continue reading from the same file-descriptor that defined in the loop header ( $input in your case )
ssh (or sshpass) read data from stdin (but in your case from file descriptor $input). And here is the point that hide the things as we didn't exect "ssh" to read the data.
Just to understand the problem you could have same strange experience for example using commands like "ffmpeg" or "mplayer" in while loop. Mplayer and ffmpeg use the keyboards while they are running, so they will consume all the the file-descriptor.
Another good and funny example :
#!/bin/bash
{
echo first
for ((i=0; i < 16384; i++)); do echo "testing"; done
echo "second"
} > test_file
while IFS= read -r line
do
echo "Read $line"
cat | uptime > /dev/null
done < test_file
At first part we write 1st line : first
16384 lines : testing
then last line : second
16384 lines "testing" are equal to 128Kb buffer
At the second part, the command "cat | uptime" will consume exactly 128Kb buffer, so our script will give
Read first
Read second
As solution, as you did, we could use "for" loop.
Or use "ssh -n"
Or playing with some file descriptor - you could find the example in the link that I gave.

Why does this command not take input from file inspite of rediection?

On executing, the cisco anyconnect VPN client takes the VPN IP, password, and some other inputs from the terminal. However, instead of typing it every time, I wrote down the values in a file and tried to redirect the file into the vpn client command.
/opt/cisco/anyconnect/bin/vpn < vpndetails.txt
However, it seems that the command ignores the file redirection and still prompts for input. How is it possible? Does the code read from some other file-descriptor (not 0) and still reads it from the terminal? Is it possible?
Note: I know it isn't a good practice to store your passwords in a file, but I don't care for now.
The question "Is it possible" has the answer "yes".
The code for the anyconnect vpn probably reads /dev/tty, as explained in the comments by Chepner e.a.As a fun exercise, try this script:
#! /bin/sh
read -p "STDIN> " a
read -p "TERMINAL> " b < /dev/tty
read -p "STDIN> " c
echo "Read $a and $c from stdio and $b from the terminal"
and, for example, ls / | bash this_script.sh.
However, if you wish to use Cisco Autoconnect without passwords, you should investigate the Always On with Trusted Network detection feature and user certificates.
Writing to /dev/tty in the hope that it will be picked-up by the script does not work:
ljm#verlaine[tmp]$ ls | bash test.sh &
[3] 10558
ljm#verlaine[tmp]$ echo 'plop' > /dev/tty
plop
[3]+ Stopped ls | bash test.sh
ljm#verlaine[tmp]$ fg
ls | bash test.sh
(a loose enter is given)
Read a_file and b_file from stdio and from the terminal

Shell Script - While Loop / File Reading not working

I have a requirement which should address following points.
I have a file which contains list of IP addresses,I want to read line by line.
For each IP I need to push following commands using SSH (all are Mikrotik devices)
/ radius add service=login address=172.16.0.1 secret=aaaa
/ user aaa set use-radius=yes
Following is my code.
#!/bin/bash
filename="branch"
while IFS= read line; do
echo ${line//}
line1=${line//}
ok='#'
line3=$ok$line1
sshpass -p abc123 ssh -o StrictHostKeyChecking=no admin$line3 / radius add service=login address=172.16.0.1 secret=aaaa
sleep 3
sshpass -p abc123 ssh -o StrictHostKeyChecking=no admin$line3 / user aaa set use-radius=yes
sleep 3
echo $line3
echo $line
done <"$filename"
Branch text file:
192.168.100.1
192.168.101.2
192.168.200.1
Issue: What ever the changes I am doing While loop is only run once.
Troubleshooting/Observations:
Without the SSH command if I run the While loop to read the file " branch " it work fine.
The problem is that a program in the loop also reads data on standard input. This will consume the 2nd and subsequent lines of what's in "$filename".
On the next iteration of the loop, there's nothing left to read and the loop terminates.
The solution is to identify the command reading stdin, probably sshpass and change it to leave stdin alone. The answer by Cyrus shows one way to do that for ssh. If that doesn't work, try
sshpass [options and arguments here] < /dev/null
Another solution is to replace the while with a for loop. This works as long as the branch file only contains IP addresses:
for ip in $(cat branch); do
echo $ip
...
sshpass ...
done

Bash, stdout redirect of commands like scp

I have a bash script with some scp commands inside.
It works very well but, if I try to redirect my stdout with "./myscript.sh >log", only my explicit echos are shown in the "log" file.
The scp output is missing.
if $C_SFTP; then
scp -r $C_SFTP_USER#$C_SFTP_HOST:$C_SOURCE "$C_TMPDIR"
fi
Ok, what should I do now?
Thank you
scp is using interactive terminal in order to print that fancy progress bar. Printing that output to a file does not make sense at all, so scp detects when its output is redirected to somewhere else other than a terminal and does disable this output.
What makes sense, however, is redirect its error output into the file in case there are errors. You might want to disable standard output if you want.
There are two possible ways of doing this. First is to invoke your script with redirection of both stderr and stdout into the log file:
./myscript.sh >log 2>&1
Second, is to tell bash to do this right in your script:
#!/bin/sh
exec 2>&1
if $C_SFTP; then
scp -r $C_SFTP_USER#$C_SFTP_HOST:$C_SOURCE "$C_TMPDIR"
fi
...
If you need to check for errors, just verify that $? is 0 after scp command is executed:
if $C_SFTP; then
scp -r $C_SFTP_USER#$C_SFTP_HOST:$C_SOURCE "$C_TMPDIR"
RET=$?
if [ $RET -ne 0 ]; then
echo SOS 2>&1
exit $RET
fi
fi
Another option is to do set -e in your script which tells bash script to report failure as soon as one of commands in scripts fails:
#!/bin/bash
set -e
...
Hope it helps. Good luck!
You cat simply test your tty with:
[ ~]#echo "hello" >/dev/tty
hello
If that works, try:
[ ~]# scp <user>#<host>:<source> /dev/tty 2>/dev/null
This has worked for me...
Unfortunately SCP's output can't simply be redirected to stdout it seems.
I wanted to get the average transfer speed of my SCP transfer, and the only way that I could manage to do that was to send stderr and stdout to a file, and then to echo the file to stdout again.
For example:
#!/bin/sh
echo "Starting with upload test at `date`:"
scp -v -i /root/.ssh/upload_test_rsa /root/uploadtest.tar.gz speedtest#myhost:/home/speedtest/uploadtest.tar.gz > /tmp/scp.log 2>&1
grep -i bytes /tmp/scp.log
rm -f /tmp/scp.log
echo "Done with upload test at `date`."
Which would result in the following output:
Starting with upload test at Thu Sep 20 13:04:44 SAST 2012:
Transferred: sent 10191920, received 5016 bytes, in 15.7 seconds
Bytes per second: sent 650371.2, received 320.1
Done with upload test at Thu Sep 20 13:05:04 SAST 2012.
I found a rough solution for scp:
$ scp -qv $USER#$HOST:$SRC $DEST
According to the scp man page, -q (quiet) disables the progress meter, as well as disabling all other output. Add -v (verbose) as well, you get heaps of output... and the progress meter is still disabled! Disabling the progress meter allows you to redirect the output to a file.
If you don't need all the authentication debug output, redirect the output to stdout and grep out the bits you don't want:
$ scp -qv $USER#$HOST:$SRC $DEST 2>&1 | grep -v debug
Final output is something like this:
Executing: program /usr/bin/ssh host myhost, user (unspecified), command scp -v -f ~/file.txt
OpenSSH_6.0p1 Debian-4, OpenSSL 1.0.1e 11 Feb 2013
Warning: Permanently added 'myhost,10.0.0.1' (ECDSA) to the list of known hosts.
Authenticated to myhost ([10.0.0.1]:22).
Sending file modes: C0644 426 file.txt
Sink: C0644 426 file.txt
Transferred: sent 2744, received 2464 bytes, in 0.0 seconds
Bytes per second: sent 108772.7, received 97673.4
Plus, this can be redirected to a file:
$ scp -qv $USER#$HOST:$SRC $DEST 2>&1 | grep -v debug > scplog.txt

Resources