This is my first question here :)
I'm running into an issue while running a remote SSH command. The issue is that the output of the command comes back in a string without LF to delimit the lines. I checked the output of Hive by itself and it outputs the LFs normally, they are missing when I run the command from a remote host. Is there any way to maintain the output of the remote host?
Here's a snippet of the code:
#!/bin/bash
partition="2013-04-02"
query=$(cat <<EOF
set mapred.job.name="Unique IP - $partition";
select
distinct
src
,dst
,service
,proto
,action
from
logs
where
src rlike "^10\."
and src not rlike "^10\.90\."
and src not rlike "^10[0-9]\."
and src not rlike "\.5[2-8]$"
and dst <> ""
and day="$partition";
EOF)
ssh=`ssh user#hadoop "hive -S -e '$query' 2>&1"
echo $ssh > output.tsv
It's got the newline(s) in there, but you're not seeing them because you don't have double-quotes around the parameter expansion in your echo command.
echo "$ssh" >output.tsv
ETA: The $(...) syntax for command substitution is generally much better than the `...` syntax, as it nests and works sanely with nested quotation marks. And while it doesn't matter in this case, it's good to get in the habit of quoting command substitutions as well:
ssh="$(ssh user#hadoop "hive -S -e '$query' 2>&1")"
Note that the double-quotes inside the $(..) don't cause any problems with the outer quotation marks.
Finally, if all you're doing with this value is echoing it into a file, you can skip the variable and just redirect it there in the first place:
ssh user#hadoop "hive -S -e '$query' 2>&1" >output.tsv
or just
ssh user#hadoop "hive -S -e '$query' " >&output.tsv
The backticks around ssh … are converting your newlines to spaces because they are supposed to.
Try
ssh user#hadoop "hive -S -e '$query' 2>&1" > output.tsv
Related
This question already has answers here:
Passing external shell script variable via ssh
(2 answers)
Closed 6 years ago.
I have a script parsing a list of servers, looking for some stuff and executing commands if the circumstances are correct.
The main server is connecting to them via ssh, executing all commands that are in the EOF statement:
#!/bin/bash
# parsing servers
# defining one local variable $VAR
ssh -T -p 1234 root#"server-ip" "$variable" << 'EOF'
# doing some stuff...
var_result=$(mysql -hhost -uuser '-ppasswort' -Ddatabase -N -e "SELECT something FROM somewhere WHERE value=$VAR;")
EOF
I know the variable can pass through if I remove the single quotes from the EOF, but if i do so the mysql statements wont work and everything breaks.
I know there are ways to transmit a variable, but things with ";" between options wont work for me ( the script tries to execute it as a command )
Any ideas?
Use printf %q to escape content in an eval-safe form; after doing so, you can pass them on the command line of the remote shell, and retrieve them via $1, $2, etc. within the remote script:
# put contents of $VAR into $var_str in a format that a shell can interpret
printf -v var_str %q "$VAR"
# v- pass the value on the shell command line
# | v- keep escaping the heredoc securely
# | |
ssh -T -p 1234 root#"$host" "bash -s $var_str" <<'EOF'
# retrieve it off the shell command line
var=$1
# ...and use it as you like thereafter.
echo "Remotely using $var"
EOF
How about using EOF without the quote and making the mysql command work:
#!/bin/bash
# parsing servers
# defining one local variable $VAR
VAR=something
ssh -T -p 1234 root#"server-ip" <<EOF
# doing some stuff...
var_result=\$(mysql -hhost -uuser '-ppasswort' -Ddatabase -N -e "SELECT something FROM somewhere WHERE value=$VAR;")
EOF
As Charles Duffy stated, this may produce some security risk.
Another way is to wrap all your codes around single quote:
#!/bin/bash
# parsing servers
# defining one local variable $VAR
ssh -T -p 1234 root#"server-ip" '
# doing some stuff...
var_result=$(mysql -hhost -uuser "-ppasswort" -Ddatabase -N -e "SELECT something FROM somewhere WHERE value='"$VAR"';")
'
In this case you will have to be careful what you substitute your variables for. Better use Charles Duffys' method if you should be concerned about it.
When I stumble across an evil web site that I want blocked from corporate access, I edit my named.conf file on my bind server and then update my proxy server blacklist file. I'd like to automate this somewhat with a bash script. Say my script is called "evil-site-block.sh" and contains the following:
ssh root#192.168.0.1 'echo "#date added $(date +%m/%d/%Y)" >> /var/named/chroot/etc/named.conf; echo "zone \"$1\" { type master; file \"/etc/zone/dummy-block\"; };" >> /var/named/chroot/etc/named.conf'
It is then run as
$ evil-site-block.sh google.com
When I look at the contents of named.conf on the remote machine I see:
#date added 09/16/2014
zone "" { type master; file "/etc/zone/dummy-block"; };
What I can't figure out is how to pass "google.com" as $1.
First off, you don't want this to be two separately redirected echo statements -- doing that is both inefficient and means that the lines could end up not next to each other if something else is appending at the same time.
Second, and much more importantly, you don't want the remote command that's run to be something that could escape its quotes and run arbitrary commands on your server (think of if $1 is '$(rm -rf /)'.spammer.com).
Instead, consider:
#!/bin/bash
# ^ above is mandatory, since we use features not found in #!/bin/sh
printf -v new_contents \
'# date added %s\nzone "%s" { type master; file "/etc/zone/dummy-block"; };\n' \
"$(date +%m/%d/%Y)" \
"$1"
printf -v remote_command \
'echo %q >>/var/named/chroot/etc/named.conf' \
"$new_contents"
ssh root#192.168.0.1 bash <<<"$remote_command"
printf %q escapes data such that an evaluation pass in another bash shell will evaluate that content back to itself. Thus, the remote shell will be guaranteed (so long as it's bash) to interpret the content correctly, even if the content attempts to escape its surrounding quotes.
Your problem: Your entire command is put into single quotes – obviously so that bash expressions are expanded on the server and not locally.
But this also applies to your $1.
Simple solution: “Interupt” the quotation by wrapping your local variable into single quotes.
ssh root#192.168.0.1 'echo "#date added $(date +%m/%d/%Y)" >> /var/named/chroot/etc/named.conf; echo "zone \"'$1'\" { type master; file \"/etc/zone/dummy-block\"; };" >> /var/named/chroot/etc/named.conf'
NB: \"$1\" → \"'$1'\".
NOTE: This solution is a simple fix for the one-liner as posted in the question above. If there's the slightest chance that this script is executed by other people, or it could process external output of any kind, please have a look at Charles Duffy's solution.
This question already has answers here:
Passing external shell script variable via ssh
(2 answers)
Closed 6 years ago.
I have a script parsing a list of servers, looking for some stuff and executing commands if the circumstances are correct.
The main server is connecting to them via ssh, executing all commands that are in the EOF statement:
#!/bin/bash
# parsing servers
# defining one local variable $VAR
ssh -T -p 1234 root#"server-ip" "$variable" << 'EOF'
# doing some stuff...
var_result=$(mysql -hhost -uuser '-ppasswort' -Ddatabase -N -e "SELECT something FROM somewhere WHERE value=$VAR;")
EOF
I know the variable can pass through if I remove the single quotes from the EOF, but if i do so the mysql statements wont work and everything breaks.
I know there are ways to transmit a variable, but things with ";" between options wont work for me ( the script tries to execute it as a command )
Any ideas?
Use printf %q to escape content in an eval-safe form; after doing so, you can pass them on the command line of the remote shell, and retrieve them via $1, $2, etc. within the remote script:
# put contents of $VAR into $var_str in a format that a shell can interpret
printf -v var_str %q "$VAR"
# v- pass the value on the shell command line
# | v- keep escaping the heredoc securely
# | |
ssh -T -p 1234 root#"$host" "bash -s $var_str" <<'EOF'
# retrieve it off the shell command line
var=$1
# ...and use it as you like thereafter.
echo "Remotely using $var"
EOF
How about using EOF without the quote and making the mysql command work:
#!/bin/bash
# parsing servers
# defining one local variable $VAR
VAR=something
ssh -T -p 1234 root#"server-ip" <<EOF
# doing some stuff...
var_result=\$(mysql -hhost -uuser '-ppasswort' -Ddatabase -N -e "SELECT something FROM somewhere WHERE value=$VAR;")
EOF
As Charles Duffy stated, this may produce some security risk.
Another way is to wrap all your codes around single quote:
#!/bin/bash
# parsing servers
# defining one local variable $VAR
ssh -T -p 1234 root#"server-ip" '
# doing some stuff...
var_result=$(mysql -hhost -uuser "-ppasswort" -Ddatabase -N -e "SELECT something FROM somewhere WHERE value='"$VAR"';")
'
In this case you will have to be careful what you substitute your variables for. Better use Charles Duffys' method if you should be concerned about it.
I am using the below command on the local machine and it gives me the expected result:
sed -n 's/^fname\(.*\)".*/\1/p' file.txt
When I use the same command(only changed ' to ") to a same file present in the remote system, I do not get any output.
ssh remote-system "sed -n "s/^fname\(.*\)".*/\1/p" file.txt"
Please help me to get this corrected. Thanks for your help.
" and ' are different things in bash, and they are not interchangeable (they're not interchangeable in many languages, however the differences are more subtle) The single quote means 'pretend everything inside here is a string'. The only thing that will be interpreted is the next single quote.
The double quote allows bash to interpret stuff inside
For example,
echo "$TERM"
and
echo '$TERM'
return different things.
(Untested) you should be able to use single quotes and escape the internal single quotes :
ssh remote-system 'sed -n \'s/^fname(.)"./\1/p\' file.txt'
Looks like you can send a single quote with the sequence '"'"' (from this question)
so :
ssh remote-machine 'sed -n '"'"'s/^fname\(.*\)".*/\1/p'"'"' file.txt'
This runs on my machine if I ssh into localhost, there's no output because file.txt is empty, but it's a proof-of-concept.
Or - can you do the ssh session interactively/with a heredoc?
ssh remote-system
[sed command]
exit
or (again untested, look up heredocs for more info)
ssh remote-system <<-EOF
[sed command]
EOF
I'm facing a small problem here, I want to pass a string containing whitespaces , to another program such that the whole string is treated as a command line argument.
In short I want to execute a command of the following structure through a bash shell script:
command_name -a arg1 -b arg2 -c "arg with whitespaces here"
But no matter how I try, the whitespaces are not preserved in the string, and is tokenized by default. A solution please,
edit: This is the main part of my script:
#!/bin/bash
#-------- BLACKRAY CONFIG ---------------#
# Make sure the current user is in the sudoers list
# Running all instances with sudo
BLACKRAY_BIN_PATH='/opt/blackray/bin'
BLACKRAY_LOADER_DEF_PATH='/home/crozzfire'
BLACKRAY_LOADER_DEF_NAME='load.xml'
BLACKRAY_CSV_PATH='/home/crozzfire'
BLACKRAY_END_POINT='default -p 8890'
OUT_FILE='/tmp/out.log'
echo "The current binary path is $BLACKRAY_BIN_PATH"
# Starting the blackray 0.9.0 server
sudo "$BLACKRAY_BIN_PATH/blackray_start"
# Starting the blackray loader utility
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "\"$BLACKRAY_END_POINT\"""
sudo time $BLACKRAY_INDEX_CMD -a $OUT_FILE
#--------- END BLACKRAY CONFIG ---------#
You're running into this problem because you store the command in a variable, then expand it later; unless there's a good reason to do this, don't:
sudo time $BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT" -a $OUT_FILE
If you really do need to store the command and use it later, there are several options; the bash-hackers.org wiki has a good page on the subject. It looks to me like the most useful one here is to put the command in an array rather than a simple variable:
BLACKRAY_INDEX_CMD=($BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT")
sudo time "${BLACKRAY_INDEX_CMD[#]}" -a $OUT_FILE
This avoids the whole confusion between spaces-separating-words and spaces-within-words because words aren't separated by spaces -- they're in separate elements of the array. Expanding the array in double-quotes with the [#] suffix preserves that structure.
(BTW, another option would be to use escaped quotes rather like you're doing, then run the command with eval. Don't do this; it's a good way to introduce weird parsing bugs.)
Edit:
Try:
BLACKRAY_END_POINT="'default -p 8890'"
or
BLACKRAY_END_POINT='"default -p 8890"'
or
BLACKRAY_END_POINT="default\ -p\ 8890"
or
BLACKRAY_END_POINT='default\ -p\ 8890'
and
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e $BLACKRAY_END_POINT"
Original answer:
Is blackray_loader a shell script?
Here is a demonstration that you have to deal with this issue both when specifying the parameter and when handling it:
A text file called "test.txt" (include the line numbers):
1 two words
2 two words
3 two
4 words
A script called "spacetest":
#!/bin/bash
echo "No quotes in script"
echo $1
grep $1 test.txt
echo
echo "With quotes in script"
echo "$1"
grep "$1" test.txt
echo
Running it with ./spacetest "two--------words" (replace the hyphens with spaces):
No quotes in script
two words
grep: words: No such file or directory
test.txt:1 two words
test.txt:2 two words
test.txt:3 two
With quotes in script
two words
2 two words
You can see that in the "No quotes" section it tried to do grep two words test.txt which interpreted "words" as a filename in addition to "test.txt". Also, the echo dropped the extra spaces.
When the parameter is quoted, as in the second section, grep saw it as one argument (including the extra spaces) and handled it correctly. And echo preserved the extra spaces.
I used the extra spaces, by the way, merely to aid in the demonstration.
I have a suggestion:
# iterate through the passed arguments, save them to new properly quoted ARGS string
while [ -n "$1" ]; do
ARGS="$ARGS '$1'"
shift
done
# invoke the command with properly quoted arguments
my_command $ARGS
probably you need to surround the argument by double quotes (e.g. "${6}").
Following OP comment it should be "$BLACKRAY_END_POINT"
Below is my example of restarting a script via exec su USER or exec su - USER. It accommodates:
being called from a relative path or current working directory
spaces in script name and arguments
single and double-quotes in arguments, without crazy escapes like: \\"
#
# This script should always be run-as a specific user
#
user=jimbob
if [ $(whoami) != "$user" ]; then
exec su -c "'$(readlink -f "$0")' $(printf " %q" "$#")" - $user
exit $?
fi
A post on other blog saved me for this whitespaces problem: http://logbuffer.wordpress.com/2010/09/23/bash-scripting-preserve-whitespaces-in-variables/
By default, whitespaces are trimed:
bash> VAR1="abc def gh ijk"
bash> echo $VAR1
abc def gh ijk
bash>
"The cause of this behaviour is the internal shell variable $IFS (Internal Field Separator), that defaults to whitespace, tab and newline.
To preserve all contiguous whitespaces you have to set the IFS to something different"
With IFS bypass:
bash> IFS='%'
bash> echo $VAR1
abc def gh ijk
bash>unset IFS
bash>
It works wonderfully for my command case:
su - user1 -c 'test -r "'${filepath}'"; ....'
Hope this helps.