How to delay a command substitution in a shell script? - bash

Question

I have a bash script that in a nutshell generates a file via other command (works fine) and then later in the script I do a wc command on the file generated. My problem is that i'm using command substitution on the wc command and when I execute the script it is executing this substitution immediately rather than waiting for the file to be generated earlier in the script. What are my options?
The script is a shell program to run Oracle SQLLoader to load data into an Oracle Table. The SQLLoader command generates a file with rejected records and I am trying to do a word count on it. This is my code:
#!/bin/sh
# Set variables
program_name="My Program Name"
input_dir="/interface/inbound/hr_datafile"
log_dir="/interface/inbound/log"
bad_dir="/interface/inbound/hr_bad"
archive_dir="/interface/inbound/hr_archive"
input_base="Import_20171213"
input_ext="csv"
control_file="data_loader.ctl"
exit_status=0
d=`date "+%Y%m%d"`
t=`date "+%H%M"`
# Check if data file exists, count records
if [ -f ${input_dir}/${input_base}*.${input_ext} ]; then
data_file_name=`ls -1rt ${input_dir}/${input_base}*.${input_ext} | head -1 | cut -c 32-100`
data_file_base=${data_file_name%.*}
echo "Data file name: " ${data_file_name}
echo "Data file base: " ${data_file_base}
no_of_recs=`expr $(wc -l ${input_dir}/${data_file_name} | cut -c 1-8) - 1`
echo "DEBUG no_of_recs: " ${no_of_recs}
no_of_errs=0
else
echo
echo
echo
echo "----------------------------- ${program_name} ------------------------------------------"
echo "----------------------------------- Error report : ------------------------------------------------"
echo
echo "Please place your Data files in the UNIX directory => "${input_dir}
echo "${program_name} Process exiting ..."
echo
echo "---------------------------------------------------------------------------------------------------"
echo
echo
echo
echo
echo
exit 1
fi
# Run SQL*Loader
echo
echo "==> Loading Data...into MY_TABLE table"
echo
# Create a temporary control file to pass the data file name in
cat $XX_TOP/bin/${control_file} | sed "s/:FILENAME/${data_file_name}/g" > $XX_TOP/bin/${data_file_base}_temp.ctl
# NOTE: The following sqlldr format is used to "hide" the oracle user name and password
sqlldr errors=100000 skip=1 control=$XX_TOP/bin/${data_file_base}_temp.ctl data=${input_dir}/${data_file_name} log=${log_dir}/${data_file_base}.log bad=${bad_dir}/${data_file_base}.bad <<-END_SQLLDR > /dev/null
apps/`cat ${DB_SCRIPTS}/.${ORACLE_SID}apps`
END_SQLLDR
exit_status=$?
echo "DEBUG exit_status " ${exit_status}
# Remove temporary control file
rm -rf $XX_TOP/bin/${data_file_base}_temp.ctl
# Check for Errors
if [ -f ${bad_dir}/${data_file_base}.bad ]; then
echo
echo "----------------------------- ${program_name} ------------------------------------------"
echo "----------------------------------- Error report : ------------------------------------------------"
echo
grep 'Record' ${log_dir}/${data_file_base}.log > ${log_dir}/${data_file_base}.rec
grep 'ORA' ${log_dir}/${data_file_base}.log > ${log_dir}/${data_file_base}.err
paste ${log_dir}/${data_file_base}.rec ${log_dir}/${data_file_base}.err ${bad_dir}/${data_file_base}.bad
echo
echo "<---------------------------------End of Error Report---------------------------------------------->"
echo
# Count error records
no_of_errs=$(wc -l ${bad_dir}/${data_file_base}.bad | cut -c 1-8)
no_of_recs=$(expr ${no_of_recs} - ${no_of_errs})
# Remove temp files
rm ${log_dir}/${data_file_base}.rec
rm ${log_dir}/${data_file_base}.err
rm ${bad_dir}/${data_file_base}.bad
else
echo "Bad File not found at ${bad_dir}/${data_file_base}.bad"
fi
if (( ${no_of_errs} > 0 )); then
echo "Error found in data file..."
exit_status=1
else
# Archive the data file if no errors
mv ${input_dir}/${data_file_name} ${archive_dir}/${data_file_base}_$d"_"$t.${input_ext}
echo "Data file archived to ${archive_dir}"
exit_status=0
fi
echo
echo
echo
echo "----------------------------- ${program_name} ------------------------------------------"
echo
echo "Total records errored out :" ${no_of_errs}
echo "Total records successfully read :" ${no_of_recs}
echo "---------------------------------------------------------------------------------------------------"
echo
# Final Exit Status
if [ ${exit_status} -eq 1 ]; then
echo "==> Exiting process...Status : ${exit_status}"
exit 1
fi
The file referenced in the if condition check is the file generated, so I'm checking if the file exists and then running the wc on it. I know that it's executing prematurely because this wc error appears in my script output before it should:
Data file name: Import_20171213.csv
Data file base: Import_20171213
DEBUG no_of_recs: 27
==> Loading Data...into MY_TABLE table
wc: 0653-755 Cannot open /interface/inbound/hr_bad/Import_20171213.bad.
Username:
SQL*Loader: Release 10.1.0.5.0 - Production on Thu Dec 21 12:42:39 2017
Copyright (c) 1982, 2005, Oracle. All rights reserved.
Commit point reached - logical record count 27
Program exited with status 2
In the code, the wc command on the .bad file is performed after the sqlldr section, yet the log shows the error occurring before sqlldr is invoked.
Any ideas would be much appreciated! Thanks!

Answer 1

You aren't seeing all of the output you expect - anything after the call to SQL*Loader is missing - and the error from wc is coming in the wrong place, before instead of after the Username; prompt. But it shouldn't be erroring giving the construct you've used - if the file doesn't exist the if test should stop that line being reach.
The issue is that it isn't seeing the end of the heredoc. All of the commands beyond the start of the heredoc are being executed as part of the heredoc processing but aren't going anywhere or being displayed to the terminal (as expected), and something in the way that is all being evaluated is causing the wc to run unexpectedly, even though the file doens't exist. You're seeing the stderr output from that.
And that is all happening before SQL*Loader starts, so you see the Username: prompt from that afterwards.
From comments it seems the heredoc-ending END_SQLLDR is really indented in your actual code, which isn't reflected in the posted question. As you used the <<- heredoc form that implies it's indented with spaces rather than tabs. That is causing that to not be recognised as the end of the here doc.
From The Linux Documentation Project:
The closing limit string, on the final line of a here document, must start in the first character position. There can be no leading whitespace. Trailing whitespace after the limit string likewise causes unexpected behavior. The whitespace prevents the limit string from being recognized.
Removing the whitespace so that is the first thing on the line will fix it.