This question already has answers here:
here-document gives 'unexpected end of file' error
(10 answers)
Closed 6 years ago.
I have the following file and I have chmod a+x on the file.
When I try to run it, I get a line 75: syntax error: unexpected end of file. What is the error with my script? What do I need to do to fix it?
#!/bin/sh
# log directory for ascp transfer
logdirectory=/tmp/log20079
# log for this script
baselog=$logdirectory/sent4files.log
#directory of where the files to be transferred are located
basedirectory=/tmp/test20079/
#remote host data
REMOTE_DIR=/Upload
REMOTE_HOST=xxx
REMOTE_USER=xxx
# extensions of file
FEATURE_EXT=feature
KEYART_EXT=keyart
TRAILER_EXT=trailer
METADATA_EXT=metadata
# files to be excluded, must match exclude_file_suffix
COMPLETE_EXT=complete
SENT_EXT=sent
# file to send on completion
FILE_ON_COMPLETE="$basedirectory"delivery.complete
if [ "$TYPE" == "File" ]; then
if [ "$STARTSTOP" == "Stop" ]; then
if [ "$STATE" == "success" ]; then
filename=$(basename "$FILE")
extension="${filename##*.}"
# check the extension here, and create files ending in .sent as a flag
case "$extension" in
"$FEATURE_EXT")
cat > "$basedirectory$FEATURE_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Feature sent" >> $baselog
;;
"$KEYART_EXT")
cat > "$basedirectory$KEYART_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Keyart sent" >> $baselog
;;
"$TRAILER_EXT")
cat > "$basedirectory$TRAILER_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Trailer sent" >> $baselog
;;
"$METADATA_EXT")
cat > "$basedirectory$METADATA_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Metadata sent" >> $baselog
;;
esac
# check that all four files exists
if [ -e "$basedirectory$FEATURE_EXT.$SENT_EXT" ] && [ -e "$basedirectory$KEYART_EXT.$SENT_EXT" ] && [ -e "$basedirectory$TRAILER_EXT.$SENT_EXT" ] && [ -e "$basedirectory$METADATA_EXT.$SENT_EXT" ]; then
echo "All files sent" >> $baselog
echo>$FILE_ON_COMPLETE
# $FILE_ON_COMPLETE "$REMOTE_USER#$REMOTE_HOST:$REMOTE_DIR"
rm "$basedirectory$FEATURE_EXT.$SENT_EXT"
rm "$basedirectory$KEYART_EXT.$SENT_EXT"
rm "$basedirectory$TRAILER_EXT.$SENT_EXT"
rm "$basedirectory$METADATA_EXT.$SENT_EXT"
rm $FILE_ON_COMPLETE
fi
fi
fi
fi
Heredocs are tricky beasts to get right. If you use 'EOF' that's exactly what the closing line needs to be, with no whitespace at the front like you have.
Alternatively, you can use the <<- variant which strips off all leading tab characters from the lines in the heredoc and the closing line as per the following transcript (where <tab> is the TAB character):
pax> cat <<-'eof'
...> 1
...< 2
...> <tab>eof
...> 4
...> eof
1
2
<tab>eof
4
pax> cat <<-'eof'
...> 1
...> 2
...> <tab>eof
1
2
Using the <<- variant allows for neater files, thoug it's no good if you want to preserve leading tabs of course. From the bash manpage:
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.
Of course, if you're just wanting to use those files as flag files, there's a better way than cat with a heredoc. Just use:
touch "$basedirectory$FEATURE_EXT.$SENT_EXT"
This will create the file if it doesn't exist and update the modification time if it does, just like the cat but without messing about with heredocs. It won't empty the file but, if you need that for some reason:
rm -f "$basedirectory$FEATURE_EXT.$SENT_EXT"
touch "$basedirectory$FEATURE_EXT.$SENT_EXT"
will do the trick.
However, since the heredoc does actually output a single empty line (one \n character), you can opt for:
echo >"$basedirectory$FEATURE_EXT.$SENT_EXT"
instead.
The end of your heredocs EOF can not have any whitespace in front of them. Remove those spaces.
Related
this is a bug I have found nothing about after a relentless search
I'm trying to run a bootstrap file in an EC2 instance, part of an EMR cluster v6.4.0. As the bootstrap action takes longer than 5 minutes, we execute it as a subprocess of the form
#!/bin/bash
var="var"
cat << EOF > ~/bootstrap.sh
intra="intra"
echo $var
echo $intra
EOF
/bin/bash ~/bootstrap.sh
exit 0
But the var "intra" is never set, and the bootstrap action returns the error line n: intra: unbound variable
If you execute that script the "intra" var is not printed.
Why can't I assign variables in a subprocess? Thank you!
When using that type of heredoc (<<WORD), you must escape literal $ characters using \$. Same goes for the backtick character (`):
#!/bin/bash
var="var"
cat << EOF > ~/bootstrap.sh
intra="intra"
echo $var
echo \$intra
EOF
/bin/bash ~/bootstrap.sh
exit 0
Another way of generating an equivalent bootstrap script is to use the literal heredoc form <<'WORD':
#!/bin/bash
var="var"
# This line will be inserted as-is without variable and subshell expansion:
cat << 'EOF1' > ~/bootstrap.sh
intra="intra"
EOF1
# We will allow this ONE line to expand.
cat << EOF2 >> ~/bootstrap.sh
echo $var
EOF2
# Back to literal insertions, no escaping necessary.
cat << 'EOF3' >> ~/bootstrap.sh
echo $intra
EOF3
/bin/bash ~/bootstrap.sh
exit 0
Inspecting the contents of ~/bootstrap.sh is a good place to start debugging.
I have two scripts. Script A includes script B and calls a function in script B.
The setup looks like this:
Test file - ~/file.txt
one==1.0.0
two==2.0.0
three==3.0.0
four==4.0.0
Script A - ~/script_a.sh
#!/bin/bash
source script_b.sh
func_one
Script B - ~/script_b.sh
#!/bin/bash
# Note: don't forget to change the spaces to tabs else heredoc won't work
my_user=$USER
func_two() {
# Here, I need run everything in the heredoc as user $my_user
sudo su - $my_user -s /bin/bash <<- EOF
while read -r line || [[ -n "$line" ]];
do
# **This is the problem line**
# I can confirm that all the lines are being
# read but echo displays nothing
echo "$line"
# The line below will be printed 4 times as there are 4 lines in the file of interest
echo "Test"
done < "/home/$my_user/file.txt"
EOF
}
func_one() {
func_two
}
To run
cd ~
bash script_a.sh
Question: Why is the line echo "$line" not producing any output?
The problem is that bash is substituting $line with its value (nothing) before it gets passed to su. Escaping the dollar sign should fix it. So $line should be changed to \$line in both places in script_b.sh.
How do I check if file exists in bash?
When I try to do it like this:
FILE1="${#:$OPTIND:1}"
if [ ! -e "$FILE1" ]
then
echo "requested file doesn't exist" >&2
exit 1
elif
<more code follows>
I always get following output:
requested file doesn't exist
The program is used like this:
script.sh [-g] [-p] [-r FUNCTION_ID|-d FUNCTION_ID] FILE
Any ideas please?
I will be glad for any help.
P.S. I wish I could show the entire file without the risk of being fired from school for having a duplicate. If there is a private method of communication I will happily oblige.
My mistake. Fas forcing a binary file into a wrong place. Thanks for everyone's help.
Little trick to debugging problems like this. Add these lines to the top of your script:
export PS4="\$LINENO: "
set -xv
The set -xv will print out each line before it is executed, and then the line once the shell interpolates variables, etc. The $PS4 is the prompt used by set -xv. This will print the line number of the shell script as it executes. You'll be able to follow what is going on and where you may have problems.
Here's an example of a test script:
#! /bin/bash
export PS4="\$LINENO: "
set -xv
FILE1="${#:$OPTIND:1}" # Line 6
if [ ! -e "$FILE1" ] # Line 7
then
echo "requested file doesn't exist" >&2
exit 1
else
echo "Found File $FILE1" # Line 12
fi
And here's what I get when I run it:
$ ./test.sh .profile
FILE1="${#:$OPTIND:1}"
6: FILE1=.profile
if [ ! -e "$FILE1" ]
then
echo "requested file doesn't exist" >&2
exit 1
else
echo "Found File $FILE1"
fi
7: [ ! -e .profile ]
12: echo 'Found File .profile'
Found File .profile
Here, I can see that I set $FILE1 to .profile, and that my script understood that ${#:$OPTIND:1}. The best thing about this is that it works on all shells down to the original Bourne shell. That means if you aren't running Bash as you think you might be, you'll see where your script is failing, and maybe fix the issue.
I suspect you might not be running your script in Bash. Did you put #! /bin/bash on the top?
script.sh [-g] [-p] [-r FUNCTION_ID|-d FUNCTION_ID] FILE
You may want to use getopts to parse your parameters:
#! /bin/bash
USAGE=" Usage:
script.sh [-g] [-p] [-r FUNCTION_ID|-d FUNCTION_ID] FILE
"
while getopts gpr:d: option
do
case $option in
g) g_opt=1;;
p) p_opt=1;;
r) rfunction_id="$OPTARG";;
d) dfunction_id="$OPTARG";;
[?])
echo "Invalid Usage" 1>&2
echo "$USAGE" 1>&2
exit 2
;;
esac
done
if [[ -n $rfunction_id && -n $dfunction_id ]]
then
echo "Invalid Usage: You can't specify both -r and -d" 1>&2
echo "$USAGE" >2&
exit 2
fi
shift $(($OPTIND - 1))
[[ -n $g_opt ]] && echo "-g was set"
[[ -n $p_opt ]] && echo "-p was set"
[[ -n $rfunction_id ]] && echo "-r was set to $rfunction_id"
[[ -n $dfunction_id ]] && echo "-d was set to $dfunction_id"
[[ -n $1 ]] && echo "File is $1"
To (recap) and add to #DavidW.'s excellent answer:
Check the shebang line (first line) of your script to ensure that it's executed by bash: is it #!/bin/bash or #!/usr/bin/env bash?
Inspect your script file for hidden control characters (such as \r) that can result in unexpected behavior; run cat -v scriptFile | fgrep ^ - it should produce NO output; if the file does contain \r chars., they would show as ^M.
To remove the \r instances (more accurately, to convert Windows-style \r\n newline sequences to Unix \n-only sequences), you can use dos2unix file to convert in place; if you don't have this utility, you can use sed 's/'$'\r''$//' file > outfile (CAVEAT: use a DIFFERENT output file, otherwise you'll destroy your input file); to remove all \r instances (even if not followed by \n), use tr -d '\r' < file > outfile (CAVEAT: use a DIFFERENT output file, otherwise you'll destroy your input file).
In addition to #DavidW.'s great debugging technique, you can add the following to visually inspect all arguments passed to your script:
i=0; for a; do echo "\$$((i+=1))=[$a]"; done
(The purpose of enclosing the value in [...] (for example), is to see the exact boundaries of the values.)
This will yield something like:
$1=[-g]
$2=[input.txt]
...
Note, though, that nothing at all is printed if no arguments were passed.
Try to print FILE1 to see if it has the value you want, if it is not the problem, here is a simple script (site below):
#!/bin/bash
file="${#:$OPTIND:1}"
if [ -f "$file" ]
then
echo "$file found."
else
echo "$file not found."
fi
http://www.cyberciti.biz/faq/unix-linux-test-existence-of-file-in-bash/
Instead of plucking an item out of "$#" in a tricky way, why don't you shift off the args you've processed with getopts:
while getopts ...
done
shift $(( OPTIND - 1 ))
FILE1=$1
I want to create a multiline file with Make, having exact content:
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
How can I tell make to output a file with this exact content somewhere?
I tried this:
define SCRIPT_CONTENT
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
endef
export SCRIPT_CONTENT
bin/script:
#echo "$$SCRIPT_CONTENT" > bin/script
This paricular solution 1) wipes $ and first char after $-es and 2) is ugly because the definition should happen outside of the particular target where it's needed :(
I also tried this:
bin/script:
#echo '
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
' > bin/script
This returns error when in make, works outside of make...
Any suggestion is very welcome!
Make wants any $ characters that should be reproduced literally to be escaped by inserting another $ in front of them.
More broadly, though, it seems like you're trying to use Make as a shell-script replacement. The more idomatic way to do this would be to put that content in a source file that you can copy to the destination, or to put it in a script that will write it into a specified destination. The Makefile then just has to invoke the copy command or the script.
With the help from this magnificent answer, I cooked up the following.
# From https://stackoverflow.com/a/8316519/874188
define \n
endef
define SCRIPT_CONTENT
#!/bin/bash
if [ "$$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$$( cat <<EOF
-Dmapred.job.tracker=$$JT
EOF
)
$${HADOOP_HOME}/bin/hadoop $$1 $$HADOOP_CONFIG_VARS $${*:2} 2>&1 | grep -v SLF4J
endef
bin/script:
echo '$(subst $(\n),\n,$(SCRIPT_CONTENT))' >$#
When testing, I found that I needed to have a semicolon at the end of the echo line when it didn't have any redirection. I can speculate that there is a built-in echo which gets invoked when there are no shell metacharacters in the command line?
Also, notice that the definition cannot contain any single quotes, and that all dollar signs have to be doubled. Maybe one or the other of these restrictions could be removed; I was unsuccessful, but then I didn't spend too much time or effort.
You cannot do this in make. Beyond what Novelocrat says regarding $, there's the fact that make is line-oriented and does not have any ability to generate a command that contains a newline character. All newlines that appear unescaped (without a backslash before them) are parsed by make as ending that recipe line, and each recipe line is sent to a different invocation of the shell. If you want the entire command to be sent as a single string to the same shell, then you must escape the newlines.
However, make will remove all backslash/newline pairs before it runs the command.
The only possible way to do this completely within make is to generate the file one line at a time, like this:
bin/script:
#echo '#!/bin/bash' > $#
#echo 'if [ "$$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi' >> $#
#echo 'export CONFIG_VARS=$$( cat <<EOF' >> $#
#echo ' -Dmapred.job.tracker=$$JT' >> $#
#echo 'EOF' >> $#
#echo ')' >> $#
#echo '$${HADOOP_HOME}/bin/hadoop $$1 $$HADOOP_CONFIG_VARS $${*:2} 2>&1 | grep -v SLF4J' >> $#
As Novelocrat says, the typical way this is done is to have the script file as a separate file and copy it where you want it, rather than generating it.
I have below script but I can't find the error. Somebody an help me ?
In concrete I split a big file in different then compress any file, move it and send by ftp rename destination filename.
Something not work :(
in line:
put ${file} ${7}.T${j}(+1)
I try to rename the file with (+1) ended new filename
Kind regards
#!/bin/bash
# configuration stuff
# ${1} absolute path file
# ${2} num_files
# ${3} output_filename
# ${4} ipMainframe ip to put files
# ${5} FTP username
# ${6} FTP password
# ${7} destination filename
if [ ! $# == 7 ]; then
#number of parameter different of two
echo "Number of parameter incorrect"
echo "Command use: LLP_split_gzip_sendFTPandTrigger.sh absolute_path_file number_of_pieces output_filename ipMainframe userFTP pwdFTP destinationFilename"
exit 1
fi
if [ -f ${1} ]; then
# If file exists
if [[ ${2} =~ ^[\-0-9]+$ ]] && (( ${2} > 0)); then
# if number of pieces is an integer > 0
#Remove old files
echo "home directory = $HOME"
CMD=`rm -f '"$HOME"/"$3"*'`
if [ $? != 0 ]; then
echo "Impossible to remove old files $home/llp_tmp* $home/"$3"* in home directory"
echo $CMD
fi
# Calculate line for every file splitted
total_lines=$(cat ${1} | wc -l)
((lines_per_file = (total_lines + ${2} - 1) / ${2}))
# Split the actual file, maintaining lines.
CMD=`split -l "$lines_per_file" "$1" "$HOME"/llp_tmp`
if [ $? != 0 ]; then
echo "SPLITTING FILE ERROR: problem to split file."
echo $CMD
exit 3
fi
#For every file splitted rename and zip it
i=1
for file in $HOME/llp_tmp*; do
CMD=`mv "$file" "$3"."$i"`
if [ $? != 0 ]; then
echo "Impossible to rename file"
echo $CMD
exit 5
fi
CMD=`gzip "$3"."$i"`
if [ $? != 0 ]; then
echo "Impossible to compress file $3.$i"
echo $CMD
exit 6
fi
i=`expr $i + 1`
done
ftp -n ${4} << EOF
j=1
user ${5} ${6}
for file in $3.*; do
put ${file} ${7}.T${j}(+1)
j=`expr $j + 1`
done
quit
else
echo "number of pieces second parameter must be more than 0."
exit 2
fi
else
echo "absolute path first paramater doesnt exist"
exit 1
fi
exit 0
You are not terminating your here document. When I run your script I get:
gash.sh: line 72: warning: here-document at line 54 delimited by end-of-file (wanted `EOF')
gash.sh: line 73: syntax error: unexpected end of file
ftp -n ${4} << EOF is the issue. Where is your here document?
The warning says it all, you don't have an EOF marker. Note that this MUST NOT BE INDENTED! The EOF must be in "column 0" and have no trailing characters, including whitespace.
Edit: It appears you want to use program constructs within a single FTP session - I don't know of a way of doing that in Bash. Perl has an easy to use FTP module where you can do it, simple example:
use strict;
use Net::FTP;
my $ftp = Net::FTP->new ("hostname");
$ftp->login ("username", "password");
$ftp->binary ();
for my $file (glob("$ENV{HOME}/llp_tmp*")) {
$ftp->put ($file);
}
$ftp->quit();
You don't need parentheses around +1.
Change it to:
put "${file}" "${7}.T${j}+1"
It's good practice to quote variables.
Another tip: Instead of j=`expr $j + 1`, you can simply use ((j++)).