bash while read loop stops after the first line (after corrections based on a StackExchange post) [duplicate] - bash

The eventual goal is to have my bash script execute a command on multiple servers. I almost have it set up. My SSH authentication is working, but this simple while loop is killing me. When I execute the while loop, reading my file for host names, it works fine when I run a
ssh $HOST "uname -a"
but when I attempt to run another ssh command,
ssh $HOST "oslevel -s"
the while loop ends early! I can't figure it out. Why would the while read do loop run perfectly fine with the first command, but not when the second is added?
I have a simple text file called hosts.list that has 4 hostnames, one per line.
$ cat hosts.list
pcced1bip04
pcced1bit04
pcced1bo02
pcced1bo04
$ cat getinfo.bash
#!/bin/bash
set -x
while read HOST
do
echo $HOST
ssh $HOST "uname -a"
#ssh $HOST "oslevel -s"
echo ""
done < hosts.list`
When it runs, it works fine. It goes through the file, line by line and gets the results of "uname -a". So everything is fine, right? (Sorry, but I turned on set -x).
$ ./getinfo.bash
+ read HOST
+ echo pcced1bip04
pcced1bip04
+ ssh pcced1bip04 'uname -a'
AIX pcced1bip04 1 6 0001431BD400
+ echo ''
+ read HOST
+ echo pcced1bit04
pcced1bit04
+ ssh pcced1bit04 'uname -a'
AIX pcced1bit04 1 6 0001431BD400
+ echo ''
+ read HOST
+ echo pcced1bo02
pcced1bo02
+ ssh pcced1bo02 'uname -a'
AIX pcced1bo02 1 6 0009FE2AD400
+ echo ''
+ read HOST
+ echo pcced1bo04
pcced1bo04
+ ssh pcced1bo04 'uname -a'
AIX pcced1bo04 1 6 0009FE2AD400
+ echo ''
+ read HOST
$
The problem occurs when I enable the line [ssh $HOST "oslevel -s"]. When I do, the script only reads the first line of the file, and then stops. Why won't it go onto the other lines?
$ ./getinfo.bash
+ read HOST
+ echo pcced1bip04
pcced1bip04
+ ssh pcced1bip04 'uname -a'
AIX pcced1bip04 1 6 0001431BD400
+ ssh pcced1bip04 'oslevel -s'
6100-06-02-1044
+ echo ''
+ read HOST
$
If I had a problem with my script, why would it be working perfectly fine with just the [ssh $HOST "uname -a"] in the while loop?

If you run commands which read from stdin (such as ssh) inside a loop, you need to ensure that either:
Your loop isn't iterating over stdin
Your command has had its stdin redirected:
...otherwise, the command can consume input intended for the loop, causing it to end.
The former:
while read -u 5 -r hostname; do
ssh "$hostname" ...
done 5<file
...which, using bash 4.1 or newer, can be rewritten with automatic file descriptor assignment as so:
while read -u "$file_fd" -r hostname; do
ssh "$hostname" ...
done {file_fd}<file
The latter:
while read -r hostname; do
ssh "$hostname" ... </dev/null
done <file
...can also, for ssh alone, can also be approximated with the -n parameter (which also redirects stdin from /dev/null):
while read -r hostname; do
ssh -n "$hostname"
done <file

Assign to an array before the loop, so that you are not using stdin for your loop variables. The ssh inside the loop can then use stdin without interfering with your loop.
readarray a < hosts.list
for HOST in "${a[#]}"; do
ssh $HOST "uname -a"
#...other stuff in loop
done

As the solution specified here use -n option for ssh or open file with a different handle:
while read -u 4 HOST
do
echo $HOST
ssh $HOST "uname -a"
ssh $HOST "oslevel -s"
echo ""
done 4< hosts.list`

maybe with python XD
#!/usr/bin/python
import sys
import Queue
from subprocess import call
logfile = sys.argv[1]
q = Queue.Queue()
with open(logfile) as data:
datalines = (line.rstrip('\r\n') for line in data)
for line in datalines:
q.put(line)
while not q.empty() :
host = q.get()
print "++++++ " + host + " ++++++"
call(["ssh", host, "uname -a"])
call(["ssh", host, "oslevel -s"])
print "++++++++++++++++++++++++++"

Related

how to run bash script interactively from url? [duplicate]

I have a simple Bash script that takes in inputs and prints a few lines out with that inputs
fortinetTest.sh
read -p "Enter SSC IP: $ip " ip && ip=${ip:-1.1.1.1}
printf "\n"
#check IP validation
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "SSC IP: $ip"
printf "\n"
else
echo "Enter a valid SSC IP address. Ex. 1.1.1.1"
exit
fi
I tried to upload them into my server, then try to run it via curl
I am not sure why the input prompt never kick in when I use cURL/wget.
Am I missing anything?
With the curl ... | bash form, bash's stdin is reading the script, so stdin is not available for the read command.
Try using a Process Substitution to invoke the remote script like a local file:
bash <( curl -s ... )
Your issue can be simply be reproduced by run the script like below
$ cat test.sh | bash
Enter a valid SSC IP address. Ex. 1.1.1.1
This is because the bash you launch with a pipe is not getting a TTY, when you do a read -p it is read from stdin which is content of the test.sh in this case. So the issue is not with curl. The issue is not reading from the tty
So the fix is to make sure you ready it from tty
read < /dev/tty -p "Enter SSC IP: $ip " ip && ip=${ip:-1.1.1.1}
printf "\n"
#check IP validation
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
echo "SSC IP: $ip"
printf "\n"
else
echo "Enter a valid SSC IP address. Ex. 1.1.1.1"
exit
fi
Once you do that even curl will start working
vagrant#vagrant:/var/www/html$ curl -s localhost/test.sh | bash
Enter SSC IP: 2.2.2.2
SSC IP: 2.2.2.2
I personally prefer source <(curl -s localhost/test.sh) option. While it is similar to bash ..., the one significant difference is how processes handled.
bash will result in a new process being spun up, and that process will evoke commands from the script.
source on the other hand will use current process to evoke commands from the script.
In some cases that can play a key role. I admit that is not very often though.
To demonstrate do the following:
### Open Two Terminals
# In the first terminal run:
echo "sleep 5" > ./myTest.sh
bash ./myTest.sh
# Switch to the second terminal and run:
ps -efjh
## Repeat the same with _source_ command
# In the first terminal run:
source ./myTest.sh
# Switch to the second terminal and run:
ps -efjh
Results should look similar to this:
Before execution:
Running bash (main + two subprocesses):
Running source (main + one subprocess):
UPDATE:
Difference in use variable usage by bash and source:
source command will use your current environment. Meaning that upon execution all changes and variable declarations, made by the script, will be available in your prompt.
bash on the other hand will be running in as a different process; therefore, all variables will be discarded when process exits.
I think everyone will agree that there are benefits and drawbacks to each method. You just have to decide which one is better for your use case.
## Test for variables declared by the script:
echo "test_var3='Some Other Value'" > ./myTest3.sh
bash ./myTest3.sh
echo $test_var3
source ./myTest3.sh
echo $test_var3
## Test for usability of current environment variables:
test_var="Some Value" # Setting a variable
echo "echo $test_var" > myTest2.sh # Creating a test script
chmod +x ./myTest2.sh # Adding execute permission
## Executing:
. myTest2.sh
bash ./myTest2.sh
source ./myTest2.sh
./myTest2.sh
## All of the above results should print the variable.
I hope this helps.

How to run for loop inside heredoc while accessing remote machine

Here is my script in which I use local variable inside a remote machine using heredoc. But the loop under the heredoc takes the first variable value only. The loop runs fine inside the heredoc but with the same values.
#!/bin/bash
prod_web=($(cat /tmp/webip.txt));
new_prod_app_private_ip=($(cat /tmp/ip.txt));
no_n=($(cat /tmp/serial.txt));
ssh -t -o StrictHostKeyChecking=no ubuntu#${prod_web[0]} -p 2345 -v << EOF
set -xv
for (( x = 0; x < '${#no_n[#]}'; x++ ))
do
sudo su
echo '${no_n[x]}'
echo '${new_prod_app_private_ip[x]}'
curl -fIkSs https://'${new_prod_app_private_ip[x]}':9002 | head -n 1
done
EOF
So, my ip.txt file contains values like:
10.0.1.0
10.0.2.0
10.0.3.0
My serial.txt file:
9
10
11
So, my loop runs for only the first IP (present in /tmp/ip.txt) in the remote machine, three times. I want to run it for all the three IPs. My remote ip is present in the file /tmp/webip.txt.
Got stuck for a long time, any help is appreciated. Is there any other solution that I can go with?
There are 2 environments. On your local machine and on the remote machine. You need to think how to transfer data/variables/state/objects/handles between these machines.
If you set something on your local machine (ie. prod_web=($(cat /tmp/webip.txt));) and then just ssh to remote host (ie. ssh user#host 'echo "${prod_web[#]}"'), the variable will not be visible/exported to the remote machine. You can:
scp the files {ip,serial}.txt and execute the whole script on the remote machine, then cleanup , ie. remove the {ip,serial}.txt files from the remote machine
pass the files {ip,serial}.txt somehow merged/joined/pasted to the stdin of the ssh and then read up stdin on the remove machine
create all the commands to run on your local machine and then pass pre-prepared commands to remote machine, like ssh .... "$(for ...; do; echo curl ...; done)"
I would go with the second option, as I like passing everything using pipes and don't like to cleanup after me - removing temporary files in case of error can be a mess.
My script would probably look like this:
#!/bin/bash
set -euo pipefail
read -r host _ <webip.txt
paste serial.txt ip.txt | ssh -t -o StrictHostKeyChecking=no -p 2345 -v ubuntu#"$host" '#!/bin/bash
set -euo pipefail
while read -r no_n ip; do
for ((i = 0; i < no_n; ++i)); do
printf "%s\n" "$no_n"
printf "%s\n" "$ip"
curl -fIkSs https://"$ip":9002 | head -n 1
done
done
'
As the remote script would become larger and less qouting friendly, I would save it into another remote_scripts.sh and execute ssh ... -m remote_scripts.sh.
I don't get what you are trying to do with that sudo su, which 100% does not do what you want.
If the no_n magic number is the number of times to execute that curl and you have xargs and you don't really care about errors, you can just do a magic and confusing oneliner:
#!/bin/bash
set -euo pipefail
read -r host _ <webip.txt
paste serial.txt ip.txt | ssh -t -o StrictHostKeyChecking=no -p 2345 -v ubuntu#"$host" 'xargs -n2 -- sh -c "seq 0 \"\$1\" | xargs -n1 -- sh -c \"curl -fIkSs https://\\\"\\\$1\\\":9002 | head -n 1\" -- \"\$2\"" --'
Preparing all the command to run maybe actually more readable and may save some nasty qouting to resolve. But this really depends on how big serial.txt and ip.txt are and how big are the commands to be executed on the remote machine, as you want to minimize the number of bytes transferred between machines.
Here the commands to run are constructed on local machine (ie. "$(...)" is passed to ssh) and executed on remote machine:
# semi-readable script, not as fast and no xargs
ssh -t -o StrictHostKeyChecking=no -p 2345 -v ubuntu#"$host" "$(paste serial.txt ip.txt | while read -r serial ip; do
seq 0 "$serial" | while read -r _; do
echo "curl -fIkSs \"https://$ip:9002\" | head -n 1"
done
done)"
HERE-doc does not expand shell commands, so:
$ cat <<EOF
> echo 1
> EOF
echo 1
but you can use command substitution $( ... ):
$ cat <<EOF
> $(echo 1)
> EOF
1

Shell Script - While Loop / File Reading not working

I have a requirement which should address following points.
I have a file which contains list of IP addresses,I want to read line by line.
For each IP I need to push following commands using SSH (all are Mikrotik devices)
/ radius add service=login address=172.16.0.1 secret=aaaa
/ user aaa set use-radius=yes
Following is my code.
#!/bin/bash
filename="branch"
while IFS= read line; do
echo ${line//}
line1=${line//}
ok='#'
line3=$ok$line1
sshpass -p abc123 ssh -o StrictHostKeyChecking=no admin$line3 / radius add service=login address=172.16.0.1 secret=aaaa
sleep 3
sshpass -p abc123 ssh -o StrictHostKeyChecking=no admin$line3 / user aaa set use-radius=yes
sleep 3
echo $line3
echo $line
done <"$filename"
Branch text file:
192.168.100.1
192.168.101.2
192.168.200.1
Issue: What ever the changes I am doing While loop is only run once.
Troubleshooting/Observations:
Without the SSH command if I run the While loop to read the file " branch " it work fine.
The problem is that a program in the loop also reads data on standard input. This will consume the 2nd and subsequent lines of what's in "$filename".
On the next iteration of the loop, there's nothing left to read and the loop terminates.
The solution is to identify the command reading stdin, probably sshpass and change it to leave stdin alone. The answer by Cyrus shows one way to do that for ssh. If that doesn't work, try
sshpass [options and arguments here] < /dev/null
Another solution is to replace the while with a for loop. This works as long as the branch file only contains IP addresses:
for ip in $(cat branch); do
echo $ip
...
sshpass ...
done

Remote SSH - executing cat and reading input

I've got the following script snippit I've written and am using:
ssh -t root#$host bash -c "'
echo \"Connected to server $host\"
echo \"Paste in data and hit Ctrl + D\"
data=$(cat)
echo \"Success!\"
echo $data
'"
It works fine without the cat line and executes in order.
With the cat line, it hangs for input before any of the echos and then when you Ctrl + D it executes the rest.
How can I get this to run back in the intended order?
EDIT:
For clarity, I'm simply attempting to get data from the local console after making the SSH connection. If I was to use a read, this works fine and prompts. But if I use "$(cat)" (which works fine locally) I have issues.
I'm attempting to take multiple lines of text in after the SSH connection. I'm using $(cat) to do this as per link, which works fine locally but doesn't appear to work in remote commands via SSH.
The reason the cat happens at the start is because you are using double quotes. The $(cat) is evaluated and executed locally before the ssh command runs. To prevent that, use single quotes instead. (I took the liberty to simplify a bit more at the same time.)
ssh -t root#$host "echo 'Connected to server $host'"';
echo "Paste in data and hit Ctrl + D"
data=$(cat)
echo "Success!"
echo "$data"'
(The first echo is in double quotes in order to allow $host to be expanded locally; then we switch to single quotes to protect the rest of the command line from local expansion.)
1- Using variables in ssh connection would be something like this:
while read pass port user ip fileinput fileoutput filetemp; do
sshpass -p$pass ssh -o 'StrictHostKeyChecking no' -p $port $user#$ip fileinput=$fileinput fileoutput=$fileoutput filetemp=$filetemp 'bash -s'<<ENDSSH1
python /path/to/f.py $fileinput $fileoutput $filetemp
ENDSSH1
done <<____HERE1
PASS PORT USER IP FILE-INPUT FILE-OUTPUT FILE-TEMP
____HERE1
So you can change this script in the way you want.
2- Can't you change your script to this way?
ssh -t root#$host bash -c "'
echo \"Connected to server $host\"
echo \"Paste in data and hit Ctrl + D\"
data=`cat <input-file>`
echo \"Success!\"
echo $data
'"
3- If you want to run some commands on remote machine try this one:
#!/bin/bash
SCRIPT='
<put your commands here>
'
while read pass ip; do
sshpass -p$pass ssh -o 'StrictHostKeyChecking no' -p <port> root#$ip "$Script"
done <<HERE
pass1 ip1
pass2 ip2
pass3 ip3
. .
. .
. .
HERE

in my bash loop over a list of some servers, if the ssh connects the bash script exits

I have a quick script to run a command on each server using ssh (i am sure there are lots of better ways to do this, but it was intended to just work quick!!). For the test1 etc, there is no server so the script continues, the script also continues if the pubkey auth fails. However if the script connects, the date is printed but the ssh loop terminates...
#!/bin/bash -x
cat <<EOF |
##file servers
test1
test2
server1
server2
EOF
while read line
do
if [ "${line:0:1}" != "#" ]; then
ssh -q -oPasswordAuthentication=no -i id_dsa user1#${line} date
fi
done
echo read line must have exited
output is like so;
+ cat
+ read line
+ '[' t '!=' '#' ']'
+ ssh -q -oPasswordAuthentication=no -i id_dsa user1#test1 date
+ read line
+ '[' t '!=' '#' ']'
+ ssh -q -oPasswordAuthentication=no -i id_dsa user1#test2 date
+ read line1
+ '[' s '!=' '#' ']'
+ ssh -q -oPasswordAuthentication=no -i id_dsa user1#server1 date
Fri Jul 9 09:04:16 PDT 2010
+ read line
+ echo read line must have exited
read line must have exited`enter code here`
something to do with the successful return of the ssh command is messing with the condition for the loop or the var... any suggestions on why?
You should pass the -n flag to ssh, to prevent it messing with stdin:
ssh -n -q -oPasswordAuthentication=no -i id_dsa user1#${line} date
I tested this with my own server and reproduced the problem, adding -n solves it. As the ssh man page says:
Redirects stdin from /dev/null
(actually, prevents reading from
stdin)
In your example, ssh must have read from stdin, which messes up your read in the loop.
I think the reason is that as ssh is being forked and exec'd in your bash script the script's standard input is being reopened so your read simultaneously terminates. Try re-crafting as follows:
for line in test1 test2 server1 server2
do
if [ "${line:0:1}" != "#" ]; then
ssh -q -oPasswordAuthentication=no -i id_dsa user1#${line} date
fi
done
or maybe run the ssh in a sub-shell like this:
( ssh -q -oPasswordAuthentication=no -i id_dsa user1#${line} date )

Resources