#!/bin/bash
export HIVE_OPTS="$HIVE_OPTS -hiveconf mapred.job.queue.name=hdmi-technology"
export HIVE_AUX_JARS_PATH=/home/hadoop/lib/HiveUDF.jar
hive -S -e 'set mapred.job.queue.name=hdmi-technology'
hive -S -e 'SELECT count(*) from testingtable2' > attachment.txt
Whenever I try to run the above shell script(count.sh) like below, I always get errors, I have no idea what wrong I am doing as I am new to shell script and I am not sure how I can add environment variables in shell script.
bash-3.00$ sh count.sh
count.sh: HIVE_OPTS= -hiveconf mapred.job.queue.name=hdmi-technology^M: is not an
identifier
Is there anything wrong I am doing in my shell script, by the way I am adding environment variables in first two lines? Any help will be appreciated.
After all the changes I did as per the below comments,
mv count.sh count,
chmod +x count,
./count
whenever I try to do this in my prompt directly export HIVE_AUX_JARS_PATH=/home/hadoop/lib/HiveUDF.jar it works fine, but whenever I try to add this in my shell script as mentioned in my question, I always get 'java.io.FileNotFoundException(File file:/home/hadoop/lib/HiveUDF.jar does not exist. Why is it so?
You're running a bash script with a sh command that is not bash. Run it with ./count.sh so the shebang line will take effect, or just say bash count.sh.
You edited your shell script on a DOS or Windows machine, which uses the CRLF pair (\r\n) instead of the Unix-style LF ('\n') line endings.
The ^M is the tell-tale -- it is a \r carriage return character.
This should fix it:
tr -d '\r' < count.sh > count.sh.fixed
mv count.sh.fixed count.sh
Another option:
sed -i 's/\r//g' count.sh
Related
I have a script that I'm trying to run but I just get the error ": command not found" whenever I try to run it. Here's what I've tried to do to fix it:
Made sure the hashbang is correct "#!/bin/bash"
Run dos2unix on the file
Run the script as scriptname.sh, ./scriptname.sh, and /bin/bash scriptname.sh
chmod 755 scriptname.sh
I still am unable to run the script. Any help is much appreciated!
This is caused by carriage returns. Here's the excerpt from the bash tag wiki:
Check whether your script or data has DOS style end-of-line characters
Use cat -v yourfile or echo "$yourvariable" | cat -v .
DOS carriage returns will show up as ^M after each line.
If you find them, delete them using dos2unix (a.k.a. fromdos) or tr -d '\r'
Make sure to check all your data, and not just the script itself.
You can use these to delete unnecessary characters:
tr -cd '[:alnum:][:blank:][:punct:]\n' < script.sh > new_script.sh
Or
tr -cd '[:graph:][:blank:]\n' < script.sh > new_script.sh
Then try new_script.sh.
I have a script that I'm trying to run but I just get the error ": command not found" whenever I try to run it. Here's what I've tried to do to fix it:
Made sure the hashbang is correct "#!/bin/bash"
Run dos2unix on the file
Run the script as scriptname.sh, ./scriptname.sh, and /bin/bash scriptname.sh
chmod 755 scriptname.sh
I still am unable to run the script. Any help is much appreciated!
This is caused by carriage returns. Here's the excerpt from the bash tag wiki:
Check whether your script or data has DOS style end-of-line characters
Use cat -v yourfile or echo "$yourvariable" | cat -v .
DOS carriage returns will show up as ^M after each line.
If you find them, delete them using dos2unix (a.k.a. fromdos) or tr -d '\r'
Make sure to check all your data, and not just the script itself.
You can use these to delete unnecessary characters:
tr -cd '[:alnum:][:blank:][:punct:]\n' < script.sh > new_script.sh
Or
tr -cd '[:graph:][:blank:]\n' < script.sh > new_script.sh
Then try new_script.sh.
I can't seem to find the difference between a script run two different ways.
Here's the script (named test.sh):
#! /bin/bash
printf "%b\n" "\u5A"
When the script is sourced:
. test.sh
> Z ## Result I want ##
When the script is run:
./test.sh
> \u5A ## Result I get ##
I want the run script to give the results of the sourced script... what setting do I need to set/change?
You are probably getting different versions of printf; the script you are sourcing from is probably a /bin/sh script, not a Bash script proper?
Shouldn't you be using \x instead of \u? printf "%b\n" "\x5A" works fine in both cases for me.
(Totally different idea here, so I'm posting it as another answer.)
Try running these at the command line:
builtin printf "%b\n" "\u5A"
/usr/bin/env printf "%b\n" "\u5A"
printf is both a shell builtin and an executable, and you may be getting different ones depending on whether you source or run the script. To find out, insert this in the script and run it each way:
type printf
While you're at it, you may as well insert this line too:
echo $SHELL
That will reveal if you're getting different shells, per tripleee.
HAHA!!! I finally traced down the problem! Read ahead if interested (leave the page if not).
These are the only command that will translate \u properly:
. ./test.sh ## Sourcing the script, hash-bang = #! /bin/sh
. ./test.bash ## Sourcing the script, hash-bang = #! /bin/bash
./test ## Running the script with no hash-bang
All of the following produce identical results in that they do NOT translate \u:
./test.sh ## Script is run from an interactive shell but in a non-interactive shell
## test.sh has first line: #! /bin/sh
/bin/sh -c "./test.sh" ## Running the script in a non-interactive sh shell
/bin/sh -lc "./test.sh" ## Running the script in a non-interactive, login sh shell
/bin/sh -c ". ./test.sh" ## Sourcing the file in a non-interactive sh shell
/bin/sh -lc ". ./test.sh" ## Sourcing the file in a non-interactive, login sh shell
## test.bash has first line: #! /bin/bash
/bin/bash -c "./test.bash" ## Running the script in a non-interactive bash shell
/bin/bash -lc "./test.bash" ## Running the script in a non-interactive, login bash shell
/bin/bash -c ". ./test.bash" ## Sourcing the file in a non-interactive bash shell
/bin/bash -lc ". ./test.bash" ## Sourcing the file in a non-interactive, login bash shell
## And from ***tripleee*** (thanks btw):
/bin/sh --norc; . ./test.sh ## Sourcing from an interactive sh shell without the ~/.bashrc file read
/bin/bash --norc; . ./test.bash ## Sourcing from an interactive bash shell without the ~/.bashrc file read
The only way to get proper translation is to run the script without a hash-bang... and I finally figured out why! Without a hash-bang my system chooses the default shell, which btw is NOT /bin/bash... it turns out to be /opt/local/bin/bash... two different versions of bash!
Finally, I removed the OSX /bin/bash [v3.2.48(1)] and replaced it with the MacPorts /opt/local/bin/bash [v4.2.10(2)] and now running the script works! It actually solved about 10-15 other problems I've had (like ${var,,}, read sN1 char, complete -EC "echo ' '", and a host of other commands I have scattered throughout my scripts, ~/.bashrc amd ~/.profile). Honestly, I really should have noticed when my scripts using associative arrays suddenly crapped out on me... how stupid can I get!?
I've been using bash v4 for a looong time now, and my Lion upgrade went and down-graded bash back to v3 (get with the program Apple!)... ugh, I feel so ashamed! Everyone still using bash v3, upgrade!! bash v4 is has many, many beautiful upgrades over version 3. Type bash --version to see what version you are running. One advantage is now bash can translate \uHEX into Unicode!
Try removing the space in the first line, I seem to recall that can cause problems. Offhand I'd guess that because of that space, you're not getting bash, but sh.
Glad you solved it. Still, you might be looking for a portable solution.
Assuming you are always using the same formatting string, we can just discard it, and use something like this;
printf () {
# Discard format string
shift
perl -CSD -le '
print map { s/^\\u//; chr(hex($_)) } #ARGV' "$#"
}
Edit to add: You would simply add this function definition at the beginning of your existing script, overriding the builtin printf. Obviously, if you also use printf for other stuff, this special-purpose replacement isn't good enough.
You could rename the function to uprintf or something, still. It merely translates a sequence of hex codes to the corresponding Unicode characters, discarding any \u prefix.
Here is the offending part of my script:
read -d '' TEXT <<'EOF'
Some Multiline
text that
I would like
in
a
var
EOF
echo "$TEXT" > ~/some/file.txt
and the error:
read: 175: Illegal option -d
I use this read -d all over the place and it works fine. Not sure why its not happy now. I'm running the script on Ubuntu 10.10
Fixes? Workarounds?
If you run sh and then try that command, you get:
read: 1: Illegal option -d
If you do it while still in bash, it works fine.
I therefore deduce that your script is not running under bash.
Make sure that your script begins with the line:
#!/usr/bin/env bash
(or equivalent) so that the correct shell is running the script.
Alternatively, if you cannot do that (because the script is not a bash one), just be aware that -d is a bash feature and may not be available in other shells. In that case, you will need to find another way.
The -d option to read is a feature unique to bash, not part of the POSIX standard (which only specifies -r and -p options to read). When you run your script with sh on Ubuntu, it's getting run with dash, which is a POSIX shell, and not bash. If you want the script to run under bash then you should run it with bash, or give it a #!/bin/bash shebang. Otherwise, it should be expected to run under any POSIX sh.
I have an sh script that contains the line
$PHP_COMMAND -r 'echo get_include_path();'
I can not edit this script, but I need the eventual command line to be (equivalent to)
php -d include_path='/path/with spaces/dir' -r 'echo get_include_path();'
How can I achieve this?
Below is a script that demonstrates the problem.
#!/bin/sh
# shell script quoting problem demonstration
# I need to be able to set a shell variable with a command with
# some options, like so
PHP_COMMAND="php -d 'include_path=/path/with spaces/dir'"
# then use PHP_COMMAND to run something in another script, like this:
$PHP_COMMAND -r 'echo get_include_path();'
# the above fails when executed. However, if you copy/paste the output
# from this line and run it in the CLI, it works!
echo "$PHP_COMMAND -r 'echo get_include_path();'"
php -d include_path='/path/with spaces/dir' -r 'echo get_include_path();'
# what's going on?
# this is also interesting
echo "\n--------------------"
# this works great, but only works if include_path doesn't need quoting
PHP_COMMAND="php -d include_path=/path/to/dir"
echo "$PHP_COMMAND -r 'echo get_include_path();'"
$PHP_COMMAND -r 'echo get_include_path();'
echo "\n--------------------"
# this one doesn't when run in the sh script, but again if you copy/paste
# the output it does work as expected.
PHP_COMMAND="php -d 'include_path=/path/to/dir'"
echo "$PHP_COMMAND -r 'echo get_include_path();'"
$PHP_COMMAND -r 'echo get_include_path();'
Script also available online: http://gist.github.com/276500
Reading your comments on other answers, I have tried to decipher what you really intended to ask:
I have an sh script that contains the line
$PHP_COMMAND -r 'echo get_include_path();'
I can not edit this script, but I need the eventual command line to be (equivalent to)
php -d include_path='/path/with spaces/dir' -r 'echo get_include_path();'
How can I achieve this?
If this accurately reflects your situation, then this is one way:
Write another script (I'll call it /some/path/to/bar.sh) to use as PHP_COMMAND. Since the variable is not quoted in the original script, you will have to make sure that the full pathname of bar.sh does not have any shell-special characters (like spaces).
/some/path/to/bar.sh
#!/bin/sh
exec php -d 'include_path=/path/with spaces/dir' ${1+"$#"}
Then, to run it, set PHP_COMMAND, and run the original script (/path/to/foo.sh):
env PHP_COMMAND=/some/path/to/bar.sh '/path/to/foo.sh'
There should be a simpler way, but one fix is to surround the entire command line with double quotes and then eval that:
PHP_COMMAND="php -d 'include_path=/path/with spaces/dir'"
eval "$PHP_COMMAND -r 'echo get_include_path();'"
Bash's printf has some special secret magic that might help. Try:
PHP_COMMAND="php -d $(printf "%q" "'include_path=/path/to/dir'")"
or some variation of that.
Edit:
I'm sorry, I should have included some explanation and an example. The %q flag causes printf to add escaping to the string so it can be reused in another command. The output of that printf would look like this (the single quotes get escaped):
\'include_path=/path/to/dir\'
This command illustrates some additional escaping:
$ printf "%q" "string ';\ text"
string\ \'\;\\\ text
what about this?
$PHP_COMMAND ' -r echo get_include_path();'