Variable not getting picked up inside EOF - shell

I'm trying to run a commands remotely by doing SSH but variable which is defined inside EOF is not getting picked up.
Variable $BASE_PATH is not getting called inside another variable which i'm defining by name FOLDER_NAME . Mentioned inside script too.
I couldn't use EOF in single quotes ('EOF') because i have to use variable from exiting shell too.
#!/usr/bin/ksh
FILE_NAME=$2
jvm_list=$1
for jvm in `echo $jvm_list`
do
ssh $jvm << EOF
echo FILE_NAME=${FILE_NAME}
export BASE_PATH="\${WEB_DOMAIN_HOME}/Server";
echo BASE_PATH=\${BASE_PATH}; ##Value of BASE_PATH is getting picked up##
export FOLDER_NAME=\`ls -1d $"{BASE_PATH}"/properties* | grep -i -v old\` ##Value of BASE_PATH is coming blank here##
echo $FOLDER_NAME
EOF
done

The construct started with << EOF acts like double-quoted string, so dollar signs are special within it, unless escaped.
Here, you escaped the $, so the shell eventually started by ssh evaluates that variable
echo BASE_PATH=\${BASE_PATH}; ##Value of BASE_PATH is getting picked up##
Here, you didn't escape it, so the evaluation happens in the outer shell:
export FOLDER_NAME=\`ls -1d $"{BASE_PATH}"/properties* ...\`
I do also suspect that the quotes there are misplaced. In Bash $".." is a locale-specific translation, and it seems to be the same in ksh. You probably don't have a translation for that, so the string should come back as is: {BASE_PATH}.
Somewhat related to this may be the backticks, since they need to be quoted too. You could use the $( ... ) form of command substitution, so you'd again, only need to think about the $.
I think this may be what you want:
ssh "$jvm" << EOF
export FOLDER_NAME=\$(ls -1d "\${BASE_PATH}"/properties* | grep -i -v old)
EOF
Sanity check:
$ foo=out; bash <<EOF
> foo=in; echo $foo
> EOF
out
$ foo=out; bash <<EOF
> foo=in; echo \$foo
> EOF
in

Related

bash invoked via ssh does not store variables

there is a problem with the invoked via ssh bash, although i have read mans about it i still can't explain the following:
Here is a script, very simple
#!/bin/bash
theUser=$1
theHost=$2
ssh -tt $theUser#$theHost 'bash' << EOF
a=1
echo 'dat '$a
exit
EOF
and here is the result:
victor#moria:~$ bash thelast.sh victor 10.0.0.8
victor#10.0.0.8's password:
a=1
echo 'dat '
exit
victor#mordor:~$ a=1
victor#mordor:~$ echo 'dat '
dat
victor#mordor:~$ exit
exit
Connection to 10.0.0.8 closed.
As you may see, the environment doesn't store the value of the variable "a" so it can't echo it, but any other commands like ls or date return the result.
So the question is what i am doing wrong and how to avoid such behavior?
p.s. i can't replace ssh -tt, but any other command may be freely replaced.
Thanks in advance
Inside the here document, the $a is expanded locally before feeding the input to the ssh command. You can prevent that by quoting the terminator after the << operator as in
ssh -tt $theUser#$theHost 'bash' << 'EOF'
$a is being expanded in the local shell, where it is undefined. In order to prevent this from happening, you should escape it:
echo "dat \$a"
Escaping the $ causes it to be passed literally to the remote shell, rather than being interpreted as an expansion locally. I have also added some double quotes, as it is good practice to enclose parameter expansions inside them.

Bash: expand variables, but not special characters

I have a bash script that creates and executes an expect script by stitching together dozens of different files containing pieces of expect code. Those files contain environment variables that need to be expanded. Example:
expect.piece:
send "command\r"
sleep $timeout
send "command argument\r"
script.sh:
#let's try it like this
eval echo $(cat expect.piece)
#or maybe like this
eval "echo \"$(cat expect.piece)\""
output:
send command\r sleep 1 send command argument\r
send commandr
sleep 1
send command argumentr
Desired otput:
send "command\r"
sleep 1
send "command argument\r"
I need a solution without sed string substitution (there is a lot of environment variables) and without modifying original expect script files. I guess it could be done line by line, but is there a more elegant solution?
Default field separator in bash is a space so set input file separator as a new line like IFS=$(echo -e '\n') before executing eval echo $(cat expect.piece) .
Final script would be :
#Storing original field separator in variable OFS
OFS=$IFS
#Setting IFS as new line using echo -e. Unfortunately IFS="\n" does not work in bash
IFS=$(echo -e '\n')
eval echo $(cat expect.piece)
#Resetting the field separator as space
IFS=$OFS
There is one another way which you can put your expect code with -c flag in the shell script as shown below.
script.sh
#Calling the expect.piece file code here
expect -c expect.piece
You can make use of the optional command line values as ,
expect -c "set count 1" myscript.exp
where the variable count will be used in the expect script file myscript.exp.
You can directly give the whole code as
expect -c "
send \"command\r\"
sleep $timeout
send \"command argument\r\"
"
Notice the use of backslash to escape the double quotes wherever needed. Single quotes can also be used. But, if you use double quotes, then only shell substitution can happen.
Please refer here to know more about the -c flag in expect.
It's not clear from your question in what context you want this output. If it's okay to embed the Expect script as a here document, what you want is trivial.
#!/bin/sh
timeout=1
cat <<____HERE
send "command\r"
sleep $timeout
send "command argument\r"
____HERE
(Maybe you can even replace the cat with expect but I'm not familiar enough with Expect to make any recommendations.)
If you need to take the input from a file, and only have a limited set of variables you want expanded, you could do that with sed.
sed "s/\$timeout/$timeout/g" file
If you need a more general solution, you might want to swich to Perl:
perl -pe 's/\$(\w+)/$ENV{$1} || "\$$1" /ge' file
but this requires you to export (or otherwise expose to Perl) the shell environment variables you want exported. (This will just not substitute any undefined variables; change the value after || if you want to change that aspect of the behavior.)
I invented the solution for this problem, it is a kludge, but it works.
expect.piece:
sleep $timeout
send "foo bar\r"
send "$(date)\r"
script.sh:
timeout=1
eval echo $(sed " \
s/\\\/\\\\\\\/g; \
s/\"/\\\\\"/g; \
s/\"/\\\\\`/\"/\\\\\`/g; \
" "expect.piece" | tr '\n' '+') | tr '+' '\n'
output:
sleep 1
send "foo bar\r"
send "Wed Feb 18 03:19:24 2015\r"
First, we need to escape all the backslashes, backticks and quotes in the file, because they will be removed during the evaluation. Then, we need to replace all the newline characters with pluses, in order to make it in a single line. After that, we evaluate that line (the evaluation will "expand" all the environment variables and command substitutions) and replace pluses back to newlines.

All arguments into files with correct quoting using "$#"

I need my bashscript to cat all of its parameters into a file. I tried to use cat for this because I need to add a lot of lines:
#!/bin/sh
cat > /tmp/output << EOF
I was called with the following parameters:
"$#"
or
$#
EOF
cat /tmp/output
Which leads to the following output
$./test.sh "dsggdssgd" "dsggdssgd dgdsdsg"
I was called with the following parameters:
"dsggdssgd dsggdssgd dgdsdsg"
or
dsggdssgd dsggdssgd dgdsdsg
I want neither of these two things: I need the exact quoting which was used on the command line. How can I achieve this? I always thought $# does everything right in regards to quoting.
Well, you are right that "$#" has the args including the whitespace in each arg. However, since the shell performs quote removal before executing a command, you can never know how exactly the args were quoted (e.g. whether with single or double quotes, or backslashes or any combination thereof--but you shouldn't need to know, since all you should care for are the argument values).
Placing "$#" in a here-document is pointless because you lose the information about where each arg starts and ends (they're joined with a space inbetween). Here's a way to see just this:
$ cat test.sh
#!/bin/sh
printf 'I was called with the following parameters:\n'
printf '"%s"\n' "$#"
$ ./test.sh "dsggdssgd" "dsggdssgd dgdsdsg"
I was called with the following parameters:
"dsggdssgd"
"dsggdssgd dgdsdsg"
Try:
#!/bin/bash
for x in "$#"; do echo -ne "\"$x\" "; done; echo
To see what's interpreted by Bash, use:
bash -x ./script.sh
or add this to the beginning of your script:
set -x
You might want add this on the parent script.

BASH script to pass variables without substitution into new script

As part of a system build script I have a script that creates various files and configurations.
However one part of the build script creates a new script that contains variables that I don't want resolved when the build script runs. Code snippet example
cat - > /etc/profile.d/mymotd.sh <<EOF
hostname=`uname -n`
echo -e "Hostname is $hostname"
EOF
I have tried all sorts of combinations of ' and " and ( and [ but I cannot get the script to send the content without substituting the values and placing the substitutes in the new script rather than the original text.
Ideas?
The easiest method, assuming you don't want anything to be substituted in the here doc, is to put the EOF marker in quotes, like this:
cat - > /etc/profile.d/mymotd.sh <<'EOF'
hostname=`uname -n`
echo -e "Hostname is $hostname"
EOF
Easiest is to escape the $
echo -e "Hostname is \$hostname"

Preserving whitespaces in a string as a command line argument

I'm facing a small problem here, I want to pass a string containing whitespaces , to another program such that the whole string is treated as a command line argument.
In short I want to execute a command of the following structure through a bash shell script:
command_name -a arg1 -b arg2 -c "arg with whitespaces here"
But no matter how I try, the whitespaces are not preserved in the string, and is tokenized by default. A solution please,
edit: This is the main part of my script:
#!/bin/bash
#-------- BLACKRAY CONFIG ---------------#
# Make sure the current user is in the sudoers list
# Running all instances with sudo
BLACKRAY_BIN_PATH='/opt/blackray/bin'
BLACKRAY_LOADER_DEF_PATH='/home/crozzfire'
BLACKRAY_LOADER_DEF_NAME='load.xml'
BLACKRAY_CSV_PATH='/home/crozzfire'
BLACKRAY_END_POINT='default -p 8890'
OUT_FILE='/tmp/out.log'
echo "The current binary path is $BLACKRAY_BIN_PATH"
# Starting the blackray 0.9.0 server
sudo "$BLACKRAY_BIN_PATH/blackray_start"
# Starting the blackray loader utility
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "\"$BLACKRAY_END_POINT\"""
sudo time $BLACKRAY_INDEX_CMD -a $OUT_FILE
#--------- END BLACKRAY CONFIG ---------#
You're running into this problem because you store the command in a variable, then expand it later; unless there's a good reason to do this, don't:
sudo time $BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT" -a $OUT_FILE
If you really do need to store the command and use it later, there are several options; the bash-hackers.org wiki has a good page on the subject. It looks to me like the most useful one here is to put the command in an array rather than a simple variable:
BLACKRAY_INDEX_CMD=($BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT")
sudo time "${BLACKRAY_INDEX_CMD[#]}" -a $OUT_FILE
This avoids the whole confusion between spaces-separating-words and spaces-within-words because words aren't separated by spaces -- they're in separate elements of the array. Expanding the array in double-quotes with the [#] suffix preserves that structure.
(BTW, another option would be to use escaped quotes rather like you're doing, then run the command with eval. Don't do this; it's a good way to introduce weird parsing bugs.)
Edit:
Try:
BLACKRAY_END_POINT="'default -p 8890'"
or
BLACKRAY_END_POINT='"default -p 8890"'
or
BLACKRAY_END_POINT="default\ -p\ 8890"
or
BLACKRAY_END_POINT='default\ -p\ 8890'
and
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e $BLACKRAY_END_POINT"
Original answer:
Is blackray_loader a shell script?
Here is a demonstration that you have to deal with this issue both when specifying the parameter and when handling it:
A text file called "test.txt" (include the line numbers):
1 two words
2 two words
3 two
4 words
A script called "spacetest":
#!/bin/bash
echo "No quotes in script"
echo $1
grep $1 test.txt
echo
echo "With quotes in script"
echo "$1"
grep "$1" test.txt
echo
Running it with ./spacetest "two--------words" (replace the hyphens with spaces):
No quotes in script
two words
grep: words: No such file or directory
test.txt:1 two words
test.txt:2 two words
test.txt:3 two
With quotes in script
two words
2 two words
You can see that in the "No quotes" section it tried to do grep two words test.txt which interpreted "words" as a filename in addition to "test.txt". Also, the echo dropped the extra spaces.
When the parameter is quoted, as in the second section, grep saw it as one argument (including the extra spaces) and handled it correctly. And echo preserved the extra spaces.
I used the extra spaces, by the way, merely to aid in the demonstration.
I have a suggestion:
# iterate through the passed arguments, save them to new properly quoted ARGS string
while [ -n "$1" ]; do
ARGS="$ARGS '$1'"
shift
done
# invoke the command with properly quoted arguments
my_command $ARGS
probably you need to surround the argument by double quotes (e.g. "${6}").
Following OP comment it should be "$BLACKRAY_END_POINT"
Below is my example of restarting a script via exec su USER or exec su - USER. It accommodates:
being called from a relative path or current working directory
spaces in script name and arguments
single and double-quotes in arguments, without crazy escapes like: \\"
#
# This script should always be run-as a specific user
#
user=jimbob
if [ $(whoami) != "$user" ]; then
exec su -c "'$(readlink -f "$0")' $(printf " %q" "$#")" - $user
exit $?
fi
A post on other blog saved me for this whitespaces problem: http://logbuffer.wordpress.com/2010/09/23/bash-scripting-preserve-whitespaces-in-variables/
By default, whitespaces are trimed:
bash> VAR1="abc def gh ijk"
bash> echo $VAR1
abc def gh ijk
bash>
"The cause of this behaviour is the internal shell variable $IFS (Internal Field Separator), that defaults to whitespace, tab and newline.
To preserve all contiguous whitespaces you have to set the IFS to something different"
With IFS bypass:
bash> IFS='%'
bash> echo $VAR1
abc def gh ijk
bash>unset IFS
bash>
It works wonderfully for my command case:
su - user1 -c 'test -r "'${filepath}'"; ....'
Hope this helps.

Resources