How to set AND expand variables in a heredoc section - bash

I have a heredoc that needs to call existing variables from the main script, and set its own variables to use later. Something like this:
count=0
ssh $other_host <<ENDSSH
if [[ "${count}" == "0" ]]; then
output="string1"
else
output="string2"
fi
echo output
ENDSSH
That doesn't work because 'output' doesn't get set to anything.
I tried using the solution from this question:
count=0
ssh $other_host << \ENDSSH
if [[ "${count}" == "0" ]]; then
output="string1"
else
output="string2"
fi
echo output
ENDSSH
It didn't work either. $output got set to "string2" because $count wasn't expanded.
How can I use a heredoc that expands variables from the parent script, and sets its own variables?

You can use:
count=0
ssh -t -t "$other_host" << ENDSSH
if [[ "${count}" == "0" ]]; then
output="string1"
else
output="string2"
fi
echo "\$output"
exit
ENDSSH
We use \$output so that it is expanded on remote host not locally.

It is better not to use stdin (such as by using here-docs) to pass commands to ssh.
If you use a command-line argument to pass your shell commands instead, you can better separate what is expanded locally and what will be executed remotely:
# Use a *literal* here-doc to read the script into a *variable*.
# Note how the script references parameter $1 instead of
# local variable $count.
read -d '' -r script <<'EOF'
[[ $1 == '0' ]] && output='zero' || output='nonzero'
echo "$output"
EOF
# The variable whose value to pass as a parameter.
# With value 0, the script will echo 'zero', otherwise 'nonzero'.
count=0
# Use `set -- '$<local-var>'...;` to pass the local variables as
# positional parameters, followed by the script code.
ssh localhost "set -- '$count'; $script"

You can escape the variables as #anubhava said, or, if you get too much variables for the escaping, you can do it in two steps:
# prepare the part which should not be expanded
# note the quoted 'EOF'
read -r -d '' commands <<'EOF'
if [[ "$count" == "0" ]]; then
echo "$count - $HOME"
else
echo "$count - $PATH"
fi
EOF
localcount=1
#use the unquoted ENDSSH
ssh me#nox.local <<ENDSSH
count=$localcount # count=1
#here will be inserted the above prepared commands
$commands
ENDSSH
will print something like:
1 - /usr/bin:/bin:/usr/sbin:/sbin

Related

bash function that either receives file name as argument or uses standard input

I want to write some wrappers around the sha1sum function in bash. From the manpage:
SHA1SUM(1) User Commands SHA1SUM(1)
NAME
sha1sum - compute and check SHA1 message digest
SYNOPSIS
sha1sum [OPTION]... [FILE]...
DESCRIPTION
Print or check SHA1 (160-bit) checksums.
With no FILE, or when FILE is -, read standard input.
How can I set up my wrapper so that it works in the same way? I.e.:
my_wrapper(){
# some code here
}
that could work both as:
my_wrapper PATH_TO_FILE
and
echo -n "blabla" | my_wrapper
I think this is somehow related to Redirect standard input dynamically in a bash script but not sure how to make it 'nicely'.
Edit 1
I program in a quite defensive way, so I use in my whole script:
# exit if a command fails
set -o errexit
# make sure to show the error code of the first failing command
set -o pipefail
# do not overwrite files too easily
set -o noclobber
# exit if try to use undefined variable
set -o nounset
Anything that works with that?
You can use this simple wrapper:
args=("$#") # save arguments into an array
set -o noclobber nounset pipefail errexit
set -- "${args[#]}" # set positional arguments from array
my_wrapper() {
[[ -f $1 ]] && SHA1SUM "$1" || SHA1SUM
}
my_wrapper "$#"
Note that you can use:
my_wrapper PATH_TO_FILE
or:
echo -n "blabla" | my_wrapper
This code works for me, put it in a file named wrapper
#!/bin/bash
my_wrapper(){
if [[ -z "$1" ]];then
read PARAM
else
PARAM="$1"
fi
echo "PARAM:$PARAM"
}
Load the function in your environment
. ./wrapper
Test the function with input pipe
root#51ce582167d0:~# echo hello | my_wrapper
PARAM:hello
Test the function with parameter
root#51ce582167d0:~# my_wrapper bybye
PARAM:bybye
Ok, so the answers posted here are fine often, but in my case with defensive programming options:
# exit if a command fails
set -o errexit
# exit if try to use undefined variable
set -o nounset
things do not work as well. So I am now using something in this kind:
digest_function(){
# argument is either filename or read from std input
# similar to the sha*sum functions
if [[ "$#" = "1" ]]
then
# this needs to be a file that exists
if [ ! -f $1 ]
then
echo "File not found! Aborting..."
exit 1
else
local ARGTYPE="Filename"
local PARAM="$1"
fi
else
local ARGTYPE="StdInput"
local PARAM=$(cat)
fi
if [[ "${ARGTYPE}" = "Filename" ]]
then
local DIGEST=$(sha1sum ${PARAM})
else
local DIGEST=$(echo -n ${PARAM} | sha1sum)
fi
}

Make a typeset function access local variable when executed remotely

I want to create a function locally, echo_a in the example, and pass it with to a remote shell through ssh, here with typeset -f. The problem is that function does not have access to the local variables.
export a=1
echo_a() {
echo a: $a
}
bash <<EOF
$(typeset -f echo_a)
echo local heredoc:
echo_a
echo
echo local raw heredoc:
echo a: $a
echo
EOF
ssh localhost bash <<EOF
$(typeset -f echo_a)
echo remote heredoc:
echo_a
echo
echo remote raw heredoc:
echo a: $a
echo
EOF
Assuming the ssh connection is automatic, running the above script gives me as output:
local heredoc:
a: 1
local raw heredoc:
a: 1
remote heredoc:
a:
remote raw heredoc:
a: 1
See how the "remote heredoc" a is empty? What can I do to get 1 there?
I tested adding quotes and backslashes everywhere without success.
What am I missing? Would something else than typeset make this work?
Thanks to #Guy for the hint, it indeed is because ssh disables by default sending the environment variables. In my case, changing the server's setting was not wanted.
Hopefully we can hack around by using compgen, eval and declare.
First we identify added variables generically. Works if variables are created inside a called function too. Using compgen is neat because we don't need to export variables explicitely.
The array diff code comes from https://stackoverflow.com/a/2315459/1013628 and the compgen trick from https://stackoverflow.com/a/16337687/1013628.
# Store in env_before all variables created at this point
IFS=$'\n' read -rd '' -a env_before <<<"$(compgen -v)"
a=1
# Store in env_after all variables created at this point
IFS=$'\n' read -rd '' -a env_after <<<"$(compgen -v)"
# Store in env_added the diff betwen env_after and env_before
env_added=()
for i in "${env_after[#]}"; do
skip=
for j in "${env_before[#]}"; do
[[ $i == $j ]] && { skip=1; break; }
done
if [[ $i == "env_before" || $i == "PIPESTATUS" ]]; then
skip=1
fi
[[ -n $skip ]] || env_added+=("$i")
done
echo_a() {
echo a: $a
}
env_added holds now an array of all names of added variables between the two calls to compgen.
$ echo "${env_added[#]}"
a
I filter out also the variables env_before and PIPESTATUS as they are added automatically by bash.
Then, inside the heredocs, we add eval $(declare -p "${env_added[#]}").
declare -p VAR [VAR ...] prints, for each VAR, the variable name followed by = followed by its value:
$ a = 1
$ b = 2
$ declare -p a b
declare -- a=1
declare -- b=2
And the eval is to actually evaluate the declare lines. The rest of the code looks like:
bash <<EOF
# Eval the variables computed earlier
eval $(declare -p "${env_added[#]}")
$(typeset -f echo_a)
echo local heredoc:
echo_a
echo
echo local raw heredoc:
echo a: $a
echo
EOF
ssh rpi_301 bash <<EOF
# Eval the variables computed earlier
eval $(declare -p "${env_added[#]}")
$(typeset -f echo_a)
echo remote heredoc:
echo_a
echo
echo remote raw heredoc:
echo a: $a
echo
EOF
Finally, running the modified script gives me the wanted behavior:
local heredoc:
a: 1
local raw heredoc:
a: 1
remote heredoc:
a: 1
remote raw heredoc:
a: 1

ssh command cannot return the result executed on remote machine

when I use ssh to execute remote file in shell script, so that I could get the result executed on remote machine, however, I met a problem which the user defined variables could not be recognized.
I created a shell script run.sh like this
#!/bin/bash
ssh jenkins#10.122.214.55 << EOF
./test.sh && ret=1 || ret=0
echo "before"
echo ${ret}
echo "after"
echo ${HOME}
exit ${ret}
EOF
the content of test.sh which is called by run.sh :
#!/bin/bash
echo "lala"
exit 1
when I call ./run.sh
it print like this
lala
before
after
/home/jenkins
Why did not it echo ${ret}? After ./run.sh is called, echo $? is 0 which is unexpected, I thought it should echo 1 here.
Thanks!
Because the variables in the heredoc are being expanded by the local script before being sent to the standard input of ssh.
In other words, your heredoc behaves similarly to
#!/bin/bash
string="echo \"before\"
echo ${ret}
echo \"after\"
echo ${HOME}
exit ${ret}
./test.sh && ret=1 || ret=0"
echo "$string" | ssh jenkins#10.122.214.55
which makes it more obvious that the string has the variables interpolated.
There are a couple ways around this: either you can escape the $ symbols (e.g. echo \${ret}) so that the intended string is passed through, or you can use a verbatim heredoc
#!/bin/bash
ssh jenkins#10.122.214.55 <<'EOF'
./test.sh && ret=1 || ret=0
echo "before"
echo ${ret}
echo "after"
echo ${HOME}
exit ${ret}
EOF
By single-quoting the delimiter, we ensure that no expansion takes place inside the heredoc.

Possible spacing issue in a bash script. Command will not run in script but will when copied from output

I've gone around and around on the quoting stuff on http://tldp.org for bash and googled until I am blue in the face. I've also tried every obvious quoting scheme for this issue, and yet nothing works.
The problem seems to be that a space inside of a quoted argument in the command run at the end of the script is being interpreted as a separator instead of as a quoted space.
Behold, here's my script (I know full well I'm a noob so comments on my style and/or uneccessary syntax is cool with me, I'll learn):
#!/bin/bash
date=`date`
args="$#"
MSEND_HOME=/home/patrol/Impact #Path to the Impact Directory
integrationName=Introscope #Name of the integration
persistEnabled=1 #1 for Yes, 0 for No
persist=""
bufDir=$MSEND_HOME/tmp/$integrationName #DO NOT CHANGE
cellName=linuxtest #Cell name to forward events to
loggingEnabled=1 #1 for Yes, 0 for No
logFile=$MSEND_HOME/log/$integrationName.$cellName.log
die () {
if [ $loggingEnabled -eq 1 ]
then
echo >>$logFile "$#"
fi
exit 1
}
[ "$#" -ge 1 ] || die "$date - At least 1 argument required, $# provided" "$#"
# This is where you would parse out your arguments and form the following
# slots as a minimum for sending an event.
class=$2
msg=\"$3\"
# Parse the first argument and assign the correct syntax
if [[ $1 == "INFORMATIONAL" ]]
then
severity=INFO
elif [[ $1 == "WARN" ]]
then
severity=WARNING
elif [[ $1 == "CRIT" ]]
then
severity=CRITICAL
else
severity=INFO
fi
#Additional slots can be set, parse them all in this variable;
#e.g., additionalSlots="slot1=value1;slot2=value2;slot3=\"value 3\""
additionalSlots=""
cmd="$MSEND_HOME/bin/msend"
cmd="$cmd -q"
cmd="$cmd -l $MSEND_HOME"
if [ $persistEnabled -eq 1 ]
then
cmd="$cmd -j $bufDir"
fi
cmd="$cmd -n $cellName"
cmd="$cmd -a $class"
cmd="$cmd -m $msg"
cmd="$cmd -r $severity"
if [ $additionalSlots ]
then
cmd="$cmd -b $additionalSlots"
fi
$cmd || die "$date - msend exited with error $? | Original arguments: $args | Command: $cmd"
#echo "msend exited with error $? | Original arguments: $args | Command: $cmd"
The script is executed like this:
./sendEvent.sh "CRIT" "EVENT" "Test Event"
The error I get from the msend executable is that the arguments are wrong, but I'm logging the command line in it's entirety to a file and when I run that logged command in the shell interactively, it works.
Here's the log output:
Tue Oct 4 20:31:29 CDT 2011 - msend exited with error 27 | Original arguments: CRIT EVENT Test Event | Command: /home/patrol/Impact/bin/msend -q -l /home/patrol/Impact -j /home/patrol/Impact/tmp/Introscope -n linuxtest -a EVENT -m "Test Event" -r CRITICAL
So if I paste /home/patrol/Impact/bin/msend -q -l /home/patrol/Impact -j /home/patrol/Impact/tmp/Introscope -n linuxtest -a EVENT -m "Test Event" -r CRITICAL and run it, it works.
If I run the script like ./sendEvent.sh "CRIT" "EVENT" "TestEvent" it works. But I need that argument to allow spaces.
I'm on the track that it's an $IFS issue or something... maybe a difference between the interactive shell and the script environment.
I'd appreciate any insight from smarter people than me!
tl;dr - My command doesn't work when run from within a script, but does when the logged command syntax is used in an interactive shell.
Short answer: see BashFAQ #50.
Long answer: When bash parses a line, it parses quote marks before doing variable substitution; as a result, when you put quotes inside a variable, they don't do what you'd expect. You're actually passing an argument list including '-m' '"Test' 'Event"' '-r' -- those double-quotes aren't around the arguments, they're in the arguments.
In this case, the best solution is to build the command in an array rather than a string. Also, get in the habbit of putting double-quotes around variables (e.g. filenames) when you use them, to prevent confusion if they contain spaces. With those changes (and a few other tweaks), here's my version of your script:
#!/bin/bash
date="$(date)" # Backquotes are confusing, use $() instead
args=("$#") # Save the args in an array rather than mushing them together in a string
MSEND_HOME=/home/patrol/Impact #Path to the Impact Directory
MSEND_HOME="$HOME/tmp" #Path to the Impact Directory
integrationName=Introscope #Name of the integration
persistEnabled=1 #1 for Yes, 0 for No
persist=""
bufDir="$MSEND_HOME/tmp/$integrationName" #DO NOT CHANGE
cellName=linuxtest #Cell name to forward events to
loggingEnabled=1 #1 for Yes, 0 for No
logFile="$MSEND_HOME/log/$integrationName.$cellName.log"
die () {
if [ $loggingEnabled -eq 1 ]
then
echo >>"$logFile" "$#"
fi
exit 1
}
[ "$#" -ge 1 ] || die "$date - At least 1 argument required, $# provided" "$#"
# This is where you would parse out your arguments and form the following
# slots as a minimum for sending an event.
class="$2" # Quotes not strictly needed here, but a good habbit
msg="$3"
# Parse the first argument and assign the correct syntax
if [[ "$1" == "INFORMATIONAL" ]]
then
severity=INFO
elif [[ "$1" == "WARN" ]]
then
severity=WARNING
elif [[ "$1" == "CRIT" ]]
then
severity=CRITICAL
else
severity=INFO
fi
#Additional slots can be set, parse them all in this array;
#e.g., additionalSlots="slot1=value1;slot2=value2;slot3=value 3" # Don't embed quotes
additionalSlots=""
cmd=("$MSEND_HOME/bin/msend") # Build the command as an array, not a string
cmd+=(-q) # Could equivalently use cmd=("${cmd[#]}" -q), but this is simpler
cmd+=(-l "$MSEND_HOME")
if [ $persistEnabled -eq 1 ]
then
cmd+=(-j "$bufDir")
fi
cmd+=(-n "$cellName")
cmd+=(-a "$class") # Possible bug: $2 and #3 aren't required, but they're getting added unconditionally
cmd+=(-m "$msg") # These should probably be conditional, like additionalSlots
cmd+=(-r "$severity")
if [ -n "$additionalSlots" ]
then
cmd+=(-b "$additionalSlots")
fi
"${cmd[#]}" || die "$date - msend exited with error $? | Original arguments:$(printf " %q" "${args[#]}") | Command:$(printf " %q" "${cmd[#]}")"
#echo "msend exited with error $? | Original arguments:$(printf " %q" "${args[#]}") | Command:$(printf " %q" "${cmd[#]}")"
I think the arg goes wrong with this assignment: cmd="$cmd -m $msg".
Change it to cmd="$cmd -m \"$msg\"".
Okay, I don't see the exact problem immediately, but I can tell you what it is; this hint should help.
Remember that the shell quoting mechanism only interprets a string once. As a result, if you're not careful, what you thought was "foo" "a" "b" is in fact "foo a b" -- that is, all one token, not three.
Run the script with bash -x which will show you at each step what the shell is actually seeing.

Bash programming with filesystem functions

I have been busy this week trying to wrap my head around a little Bash program to migrate a CMS from one server to another. The reasopn for this is because I have more tha 40 of these to do, and need to get it done in a timely manner, thus the Bash idea.
Needless to say, I have run into a couple of problems so far, but one of them has halted my development completetly, directory checking.
No I have tried a couple of methods and none of them seem to work really. The catch is that I have to check the folder on a remote server via ssh. Here my example:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
SSHRoot='ssh -i $ExSshRsa -p $ExSshPort $ExSshHost [ -d $ExRoot ] || exit 1 '
echo $SSHRoot
if [ "$SSHRoot" -eq 0 ]
then
echo "OK"
else
echo "FAIL"
fi
I get the Error: [: : integer expression expected
Does the [ or test not resturn a 0 which is numerical. ?
Passing strings as arguments to a remote host is not trivial; you need to use arrays. A test example:
declare -a cmd=(touch "file name with spaces")
printf -v escaped_cmd_str '%q ' "${cmd[#]}"
ssh localhost $escaped_cmd
ssh localhost ls # Should return "file name with spaces" on a separate line
So your case should be:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
declare -a cmd=( '[' -d "$ExRoot" ']' ) # Need to quote "[" since it's a Bash-specific symbol
printf -v escaped_cmd_str '%q ' "${cmd[#]}"
if ssh -i "$ExSshRsa" -p "$ExSshPort" "$ExSshHost" $escaped_cmd
then
echo "OK"
else
echo "FAIL"
fi
This is a rare case where using unquoted variable expansion is perfectly fine.
change the shebang to #!/bin/bash -x and look at the output...
you are storing a string in variable SSHRoot using single quotes, meaning that no variables will be expanded, i.e. a $ is still a $. Use double quotes instead, i.e. "
to store the output from a command in bash, use
var=$(cmd)
the exist status of a command is stored in the variable $?. Do a check on that after the ssh-command
you are never executing the ssh-command in your code
Great link here for bash-programming
Try the following:
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExRoot=/var/www/
echo -n "Verifying Root access $ExRoot..."
cmd="bash -c \"[ -d $ExRoot ] || exit 1\""
SSHRoot="ssh -i $ExSshRsa -p $ExSshPort $ExSshHost ${cmd}"
$SSHRoot
if [ $? -eq 0 ]
then
echo "OK"
else
echo "FAIL"
fi
The variables weren't being replaced in your SSHRoot variable as it's in single quotes. Also, you weren't passing an executable command, so that's why I use bash -c above. It will run the bash commands inside the quoted string.
$? stores the exit value of the last command, in this case the SSHRoot one.
#!/bin/bash
ExSshRsa=~/.ssh/id_rsa
ExSshPort=22
ExSshHost=localhost
ExBase='/tmp/'
ExRoot='one space/'
declare -a AExRoot
for argR in "${ExRoot[#]}"
do
ExRoot+=($(printf %q "$argR"))
done
clear
FRoot=( $ExBase${ExRoot[#]} )
echo -n "Verifying Root access $FRoot..."
SSHRootTest="bash -c \"[ -d $FRoot ] && echo 0 && exit 0 || echo 1 && exit 1\""
SSHRoot=$( ssh -i $ExSshRsa -p $ExSshPort $ExSshHost ${SSHRootTest})
if [ $? -eq 0 ]
then
echo -en "\e[1;32mOK\e[0;37;m..."
else
echo -en "\e[1;31mFAIL\e[0;37;m..."
fi
sleep 1
if [ -w $FRoot ]
then
echo -e "\e[1;32mwritable\e[0;37;m"
else
echo -e "\e[1;31mNOT writeable\e[0;37;m"
fi
echo -e "\e[0;m"
exit 0
So I have incorporated all of the suggestions so far and have one last problem, the FRoot is not getting populated by the complete array values. Other than that I think it now has the subjective approach as suggested #john-keyes, the proper expansion #frederik and the crazy space escapes #l0b0

Resources