When I run the following command in a terminal it works, but not from a script:
eval $(printf "ssh foo -f -N "; \
for port in $(cat ~/bar.json | grep '_port' | grep -o '[0-9]\+'); do \
printf "-L $port:127.0.0.1:$port ";\
done)
The error I get tells me that printf usage is wrong, as if the -L argument within quotes would've been an argument to printf itself.
I was wondering why that is the case. Am I missing something obvious?
__
Context (in case my issue is an XY problem): I want to start and connect to a jupyter kernel running on a remote computer. To do so I wrote a small script that
sends a command per ssh for the remote to start the kernel
copies via scp a configuration file that I can use to connect to the kernel from my local computer
reads the configuration file and opens appropriate ssh tunnels between local and remote
For those not familiar with jupyter, a configuration file (bar.json) looks more or less like the following:
{
"shell_port": 35932,
"iopub_port": 37145,
"stdin_port": 42704,
"control_port": 39329,
"hb_port": 39253,
"ip": "127.0.0.1",
"key": "4cd3e12f-321bcb113c204eca3a0723d9",
"transport": "tcp",
"signature_scheme": "hmac-sha256",
"kernel_name": ""
}
And so, in my command above, the printf statement creates an ssh command with all the 5 -L port forwarding for my local computer to connect to the remote, and eval should run that command. Here's the full script:
#!/usr/bin/env bash
# Tell remote to start a jupyter kernel.
ssh foo -t 'python -m ipykernel_launcher -f ~/bar.json' &
# Wait a bit for the remote kernel to launch and write conf. file
sleep 5
# Copy the conf. file from remote to local.
scp foo:~/bar.json ~/bar.json
# Parse the conf. file and open ssh tunnels.
eval $(printf "ssh foo -f -N "; \
for port in $(cat ~/bar.json | grep '_port' | grep -o '[0-9]\+'); do \
printf "-L $port:127.0.0.1:$port ";\
done)
Finally, jupyter console --existing ~/foo.json connects to remote.
As #that other guy says, bash's printf builtin barfs on printf "-L ...". It thinks you're passing it a -L option. You can fix it by adding --:
printf -- "-L $port:127.0.0.1:$port "
Let's make that:
printf -- '-L %s:127.0.0.1:%s ' "$port" "$port"
But since we're here, we can do a lot better. First, let's not process JSON with basic shell tools. We don't want to rely on it being formatting a certain way. We can use jq, a lightweight and flexible command-line JSON processor.
$ jq -r 'to_entries | map(select(.key | test(".*_port"))) | .[].value' bar.json
35932
37145
42704
39329
39253
Here we use to_entries to convert each field to a key-value pair. Then we select entries where the .key matches the regex .*_port. Finally we extract the corresponding .values.
We can get rid of eval by constructing the ssh command in an array. It's always good to avoid eval when possible.
#!/bin/bash
readarray -t ports < <(jq -r 'to_entries | map(select(.key | test(".*_port"))) | .[].value' bar.json)
ssh=(ssh foo -f -N)
for port in "${ports[#]}"; do ssh+=(-L "$port:127.0.0.1:$port"); done
"${ssh[#]}"
Related
Here is my script in which I use local variable inside a remote machine using heredoc. But the loop under the heredoc takes the first variable value only. The loop runs fine inside the heredoc but with the same values.
#!/bin/bash
prod_web=($(cat /tmp/webip.txt));
new_prod_app_private_ip=($(cat /tmp/ip.txt));
no_n=($(cat /tmp/serial.txt));
ssh -t -o StrictHostKeyChecking=no ubuntu#${prod_web[0]} -p 2345 -v << EOF
set -xv
for (( x = 0; x < '${#no_n[#]}'; x++ ))
do
sudo su
echo '${no_n[x]}'
echo '${new_prod_app_private_ip[x]}'
curl -fIkSs https://'${new_prod_app_private_ip[x]}':9002 | head -n 1
done
EOF
So, my ip.txt file contains values like:
10.0.1.0
10.0.2.0
10.0.3.0
My serial.txt file:
9
10
11
So, my loop runs for only the first IP (present in /tmp/ip.txt) in the remote machine, three times. I want to run it for all the three IPs. My remote ip is present in the file /tmp/webip.txt.
Got stuck for a long time, any help is appreciated. Is there any other solution that I can go with?
There are 2 environments. On your local machine and on the remote machine. You need to think how to transfer data/variables/state/objects/handles between these machines.
If you set something on your local machine (ie. prod_web=($(cat /tmp/webip.txt));) and then just ssh to remote host (ie. ssh user#host 'echo "${prod_web[#]}"'), the variable will not be visible/exported to the remote machine. You can:
scp the files {ip,serial}.txt and execute the whole script on the remote machine, then cleanup , ie. remove the {ip,serial}.txt files from the remote machine
pass the files {ip,serial}.txt somehow merged/joined/pasted to the stdin of the ssh and then read up stdin on the remove machine
create all the commands to run on your local machine and then pass pre-prepared commands to remote machine, like ssh .... "$(for ...; do; echo curl ...; done)"
I would go with the second option, as I like passing everything using pipes and don't like to cleanup after me - removing temporary files in case of error can be a mess.
My script would probably look like this:
#!/bin/bash
set -euo pipefail
read -r host _ <webip.txt
paste serial.txt ip.txt | ssh -t -o StrictHostKeyChecking=no -p 2345 -v ubuntu#"$host" '#!/bin/bash
set -euo pipefail
while read -r no_n ip; do
for ((i = 0; i < no_n; ++i)); do
printf "%s\n" "$no_n"
printf "%s\n" "$ip"
curl -fIkSs https://"$ip":9002 | head -n 1
done
done
'
As the remote script would become larger and less qouting friendly, I would save it into another remote_scripts.sh and execute ssh ... -m remote_scripts.sh.
I don't get what you are trying to do with that sudo su, which 100% does not do what you want.
If the no_n magic number is the number of times to execute that curl and you have xargs and you don't really care about errors, you can just do a magic and confusing oneliner:
#!/bin/bash
set -euo pipefail
read -r host _ <webip.txt
paste serial.txt ip.txt | ssh -t -o StrictHostKeyChecking=no -p 2345 -v ubuntu#"$host" 'xargs -n2 -- sh -c "seq 0 \"\$1\" | xargs -n1 -- sh -c \"curl -fIkSs https://\\\"\\\$1\\\":9002 | head -n 1\" -- \"\$2\"" --'
Preparing all the command to run maybe actually more readable and may save some nasty qouting to resolve. But this really depends on how big serial.txt and ip.txt are and how big are the commands to be executed on the remote machine, as you want to minimize the number of bytes transferred between machines.
Here the commands to run are constructed on local machine (ie. "$(...)" is passed to ssh) and executed on remote machine:
# semi-readable script, not as fast and no xargs
ssh -t -o StrictHostKeyChecking=no -p 2345 -v ubuntu#"$host" "$(paste serial.txt ip.txt | while read -r serial ip; do
seq 0 "$serial" | while read -r _; do
echo "curl -fIkSs \"https://$ip:9002\" | head -n 1"
done
done)"
HERE-doc does not expand shell commands, so:
$ cat <<EOF
> echo 1
> EOF
echo 1
but you can use command substitution $( ... ):
$ cat <<EOF
> $(echo 1)
> EOF
1
This question already has answers here:
How do I set a variable to the output of a command in Bash?
(15 answers)
Closed 4 years ago.
I have a script like that
genhash --use-ssl -s $IP -p 443 --url $URL | grep MD5 | grep -c $MD5
I want to get stream generated by genhash in a variable. How do I redirect it into a variable $hash to compare inside a conditional?
if [ $hash -ne 0 ]
then echo KO
exit 0
else echo -n OK
exit 0
fi
Use the $( ... ) construct:
hash=$(genhash --use-ssl -s $IP -p 443 --url $URL | grep MD5 | grep -c $MD5)
TL;DR
To store "abc" into $foo:
echo "abc" | read foo
But, because pipes create forks, you have to use $foo before the pipe ends, so...
echo "abc" | ( read foo; date +"I received $foo on %D"; )
Sure, all these other answers show ways to not do what the OP asked, but that really screws up the rest of us who searched for the OP's question.
The answer to the question is to use the read command.
Here's how you do it
# I would usually do this on one line, but for readability...
series | of | commands \
| \
(
read string;
mystic_command --opt "$string" /path/to/file
) \
| \
handle_mystified_file
Here is what it is doing and why it is important:
Let's pretend that the series | of | commands is a very complicated series of piped commands.
mystic_command can accept the content of a file as stdin in lieu of a file path, but not the --opt arg therefore it must come in as a variable. The command outputs the modified content and would commonly be redirected into a file or piped to another command. (E.g. sed, awk, perl, etc.)
read takes stdin and places it into the variable $string
Putting the read and the mystic_command into a "sub shell" via parenthesis is not necessary but makes it flow like a continuous pipe as if the 2 commands where in a separate script file.
There is always an alternative, and in this case the alternative is ugly and unreadable compared to my example above.
# my example above as a oneliner
series | of | commands | (read string; mystic_command --opt "$string" /path/to/file) | handle_mystified_file
# ugly and unreadable alternative
mystic_command --opt "$(series | of | commands)" /path/to/file | handle_mystified_file
My way is entirely chronological and logical. The alternative starts with the 4th command and shoves commands 1, 2, and 3 into command substitution.
I have a real world example of this in this script but I didn't use it as the example above because it has some other crazy/confusing/distracting bash magic going on also.
read hash < <(genhash --use-ssl -s $IP -p 443 --url $URL | grep MD5 | grep -c $MD5)
This technique uses Bash's "process substitution" not to be confused with "command substitution".
Here are a few good references:
http://www.linuxjournal.com/content/shell-process-redirection
http://tldp.org/LDP/abs/html/process-sub.html
http://tldp.org/LDP/abs/html/commandsub.html ☚ for comparison
I guess compatible way:
hash=`genhash --use-ssl -s $IP -p 443 --url $URL | grep MD5 | grep -c $MD5`
but I prefer
hash="$(genhash --use-ssl -s $IP -p 443 --url $URL | grep MD5 | grep -c $MD5)"
If a pipeline is too complicated to wrap in $(...), consider writing a function. Any local variables available at the time of definition will be accessible.
function getHash {
genhash --use-ssl -s $IP -p 443 --url $URL | grep MD5 | grep -c $MD5
}
hash=$(getHash)
http://www.gnu.org/software/bash/manual/bashref.html#Shell-Functions
You can do:
hash=$(genhash --use-ssl -s $IP -p 443 --url $URL)
or
hash=`genhash --use-ssl -s $IP -p 443 --url $URL`
If you want to result of the entire pipe to be assigned to the variable, you can use the entire pipeline in the above assignments.
I got error sometimes when using $(`code`) constructor.
Finally i got some approach to that here: https://stackoverflow.com/a/7902174/2480481
Basically, using Tee to read again the ouput and putting it into a variable.
Theres how you see the normal output then read it from the ouput.
is not? I guess your current task genhash will output just that, a single string hash so might work for you.
Im so neewbie and still looking for full output & save into 1 command.
Regards.
Create a function calling it as the command you want to invoke. In this case, I need to use the ruok command.
Then, call the function and assign its result into a variable. In this case, I am assigning the result to the variable health.
function ruok {
echo ruok | nc *ip* 2181
}
health=echo ruok *ip*
I wrote a small bash script in this post: How to search for a string in a text file and perform a specific action based on the result
I noticed that when I ran the script and check the logs, everything appears to be working but when I look at the Nagios UI, almost half of the servers listed in my text file did not get their notifications disabled. A revised version of the script is below:
host=/Users/bob/wsus.txt
password="P#assw0rd123"
while read -r host; do
region=$(echo "$host" | cut -f1 -d-)
if [[ $region == *sea1* ]]
then
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
else
echo "Disabling host notifications for: $host"
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah02.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
fi
done < wsus.txt >> /Users/bob/disable.log 2>&1
If i run the command against the servers having the issue manually, it does get disabled in the Nagios UI, so I'm a bit confused. FYI, I'm not well versed in Bash either so this was my attempt at trying to automate this process a bit.
1 - There is a missing double-quote before the first https occurence:
You have:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
Should be:
curl -vs -o /dev/null -d "cmd_mod=2&cmd_typ=25&host=$host&btnSubmit=Commit" "https://nagios.$region.blah.com/nagios/cgi-bin/cmd.cgi" -u "bob:$password" -k 2>&1
2 - Your first variable host is never used (overwritten inside the while loop).
I'm guessing what you were trying to do was something like:
hosts_file="/Users/bob/wsus.txt"
log_file="/Users/bob/disable.log"
# ...
while read -r host; do
# Do stuff with $host
done < $hosts_file >> $log_file 2>&1
3 - This looks suspicious to me:
if [[ $region == *sea1* ]]
Note: I haven't tested it yet, so this is my general feeling about this, might be wrong.
The $region isn't double-quoted, so make sure there could be no spaces / funny stuff happening there (but this should not be a problem inside a double-bracket test [[).
The *sea* looks like it would be expanded to match your current directory files matching this globbing. If you want to test this as a regular expression, you should use ~= operator or (my favorite for some reason) grep command:
if grep -q ".*sea.*" <<< "$region"; then
# Your code if match
else
# Your code if no match
fi
The -q keeps grep quiet
There is no need for test like [ or [[ because the return code of grep is already 0 if any match
The <<< simply redirects the right strings as the standard input of the left command (avoid useless piping like echo "$region" | grep -q ".*sea.*").
If this doesn't solve your problem, please provide a sample of your input file hosts_file as well as some output logs.
You could also try to see what's really going on under the hood by enclosing your script with set -x and set +x to activate debug/trace mode.
I'm trying to decode my password through a script while it's being run but it seems like the script is being run with a literal and the password is not processed. Is there a better way of doing this?
#!/bin/bash
MYENC="Tk9UX1RIQVRfU1RVUElEX0xPTAo="
rdesktop -u FOO -d mgmt -p 'echo $(echo $MYENC) | base64 --decode' 192.0.0.0
I also tried to just pass in a variable but that failed as well.
Try this instead:
#!/bin/bash
MYENC="Tk9UX1RIQVRfU1RVUElEX0xPTAo="
rdesktop -u FOO -d mgmt -p $(echo $MYENC | base64 --decode) 192.0.0.0
Note that I wrapped the juicy stuff echo...base64... in $(...). This is called "command substitution" - basically you're telling bash that you want the code inside the $(...) to be executed before the rest of the line, with the result substituted in its place. More info here: http://www.tldp.org/LDP/abs/html/commandsub.html
Or this
#!/bin/bash
MYENC="Tk9UX1RIQVRfU1RVUElEX0xPTAo="
rdesktop -u FOO -d mgmt -p $(base64 --decode <<< "$MYENC") 192.0.0.0
I have this shell script which ssh to other server, find few specific files(.seq files older than 50 mnts) and writes their name to another file.
#! /usr/bin/bash
while read line
do
#echo $line
if [[ $line =~ ^# ]];
then
#echo $line;
continue;
else
serverIP=`echo $line|cut -d',' -f1`
userID=`echo $line|cut -d',' -f2`
fi
done < sftp.conf
sshpass -p red32hat ssh $userID#$serverIP
cd ./perl
for files in `find -name "*.seq" -mmin +50`
do
#sshpass -p red32hat scp *.seq root#rinacac-test:/root/perl
echo $files>>abcde.txt
done
exit;
#EOF
Now problem is that when I run it.. neither it writes to abcde.txt file nor it is exiting from the remote server. when I manually execute the exit command...it exists saying "perl no such file or directory"... while I have perl sub directory in my home directory..
other thing is when I run the for loop portion of the script on the 2nd server(by directly logging into it) it is working fine and writing to abcde.txt filr...please help...
ssh takes commands either on standard input or as the last parameter. You can therefore do this (very dynamic but tricky to get the expansions right):
ssh user#host <<EOF
some
custom
commands
EOF
or this (less dynamic but can take simple parameters without escaping):
scp my_script.sh user#host:
ssh user#host './my_script.sh'