Generate multiline file with Make - bash

I want to create a multiline file with Make, having exact content:
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
How can I tell make to output a file with this exact content somewhere?
I tried this:
define SCRIPT_CONTENT
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
endef
export SCRIPT_CONTENT
bin/script:
#echo "$$SCRIPT_CONTENT" > bin/script
This paricular solution 1) wipes $ and first char after $-es and 2) is ugly because the definition should happen outside of the particular target where it's needed :(
I also tried this:
bin/script:
#echo '
#!/bin/bash
if [ "$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$( cat <<EOF
-Dmapred.job.tracker=$JT
EOF
)
${HADOOP_HOME}/bin/hadoop $1 $HADOOP_CONFIG_VARS ${*:2} 2>&1 | grep -v SLF4J
' > bin/script
This returns error when in make, works outside of make...
Any suggestion is very welcome!

Make wants any $ characters that should be reproduced literally to be escaped by inserting another $ in front of them.
More broadly, though, it seems like you're trying to use Make as a shell-script replacement. The more idomatic way to do this would be to put that content in a source file that you can copy to the destination, or to put it in a script that will write it into a specified destination. The Makefile then just has to invoke the copy command or the script.

With the help from this magnificent answer, I cooked up the following.
# From https://stackoverflow.com/a/8316519/874188
define \n
endef
define SCRIPT_CONTENT
#!/bin/bash
if [ "$$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi
export CONFIG_VARS=$$( cat <<EOF
-Dmapred.job.tracker=$$JT
EOF
)
$${HADOOP_HOME}/bin/hadoop $$1 $$HADOOP_CONFIG_VARS $${*:2} 2>&1 | grep -v SLF4J
endef
bin/script:
echo '$(subst $(\n),\n,$(SCRIPT_CONTENT))' >$#
When testing, I found that I needed to have a semicolon at the end of the echo line when it didn't have any redirection. I can speculate that there is a built-in echo which gets invoked when there are no shell metacharacters in the command line?
Also, notice that the definition cannot contain any single quotes, and that all dollar signs have to be doubled. Maybe one or the other of these restrictions could be removed; I was unsuccessful, but then I didn't spend too much time or effort.

You cannot do this in make. Beyond what Novelocrat says regarding $, there's the fact that make is line-oriented and does not have any ability to generate a command that contains a newline character. All newlines that appear unescaped (without a backslash before them) are parsed by make as ending that recipe line, and each recipe line is sent to a different invocation of the shell. If you want the entire command to be sent as a single string to the same shell, then you must escape the newlines.
However, make will remove all backslash/newline pairs before it runs the command.
The only possible way to do this completely within make is to generate the file one line at a time, like this:
bin/script:
#echo '#!/bin/bash' > $#
#echo 'if [ "$$JAVA_HOME" = "" ]; then echo "Please set JAVA_HOME"; exit 1; fi' >> $#
#echo 'export CONFIG_VARS=$$( cat <<EOF' >> $#
#echo ' -Dmapred.job.tracker=$$JT' >> $#
#echo 'EOF' >> $#
#echo ')' >> $#
#echo '$${HADOOP_HOME}/bin/hadoop $$1 $$HADOOP_CONFIG_VARS $${*:2} 2>&1 | grep -v SLF4J' >> $#
As Novelocrat says, the typical way this is done is to have the script file as a separate file and copy it where you want it, rather than generating it.

Related

wildcard * not behaving in expected way in bash script

I have a bash script as shown below. I run this in a directory containing files such as input1.inp and other files like coords_i.xyz and submission.sub in order to make some simple modifications to them:
#!/bin/bash
sed -i -e '25d' *.inp
echo "*xyz -2 2" >> *.inp
sed -n '3,7p' *_i.xyz >> *.inp
echo "Q -1 0 0 3" >> *.inp
echo "Q +1 0 0 -3" >> *.inp
echo "*" >> *.inp
sed -i -e s/"replace1"/"replace2"/g *.sub
rm *.out
If I am in this directory, and I run all the commands individually in the terminal (line by line in the script), everything works fine. However, when I try to group all these commands into the script as shown above, I get an error - essentially after the line sed -i -e '25d' *.inp, the script stops and a file called *.inp is created in my directory. If I try to run the echo command separately after that, it says the command is ambiguous (presumably because of the existence of this *.inp file).
Why don't my wildcards work the same way in the script as they did when I ran them separately and sequentially in the terminal, and what can I do so that they work properly in the script?
Using wildcards this way is hazardous; the easy advice is "don't". Evaluate them only once, and then you can check their outputs before trying to use them.
In the below, we define an assert_only_one function that stops your script when an array -- assigned from a glob -- contains less or more than exactly one element. Consequently, we're able to write code that more clearly and explicitly describes our desired behavior.
#!/usr/bin/env bash
shopt -s nullglob # Stop *.xyz evaluating to '*.xyz' if no such files exist
assert_only_one() {
local glob; glob=$1; shift
case $# in
0) echo "ERROR: No files matching $glob exist" >&2; exit 1;;
1) return 0;;
*) echo "ERROR: More than one file matching $glob exists:" >*2
printf ' %q\n' "$#" >&2
exit 1;;
esac
}
inp_files=( *.inp ); assert_only_one '*.inp' "${inp_files[#]}"
sub_files=( *.sub ); assert_only_one '*.sub' "${sub_files[#]}"
xyz_files=( *_i.xyz )
sed -i -e '25d' "${inp_files[0]}"
{
echo "*xyz -2 2"
sed -n '3,7p' "${xyz_files[#]}"
echo "Q -1 0 0 3"
echo "Q +1 0 0 -3"
echo "*"
} >>"${inp_files[0]}"
sed -i -e s/"replace1"/"replace2"/g -- "${sub_files[#]}"
rm -- *.out

$$ in shell , Linux

I have the following code from a make file, I know this creates bin folder in Home if that doesn't exist... but I couldn't understand what $$HOME/bin mean...
I googled and found $$ is to get the processid of the bash... but couldn't understand what $$HOME/bin mean... can someone please explain ?
.PHONY: home_bin
home_bin: ## Create home bin if not created
# if [[ ! -d "$$HOME/bin" ]]; then \
echo "Creating $$HOME/bin"; \
mkdir $$HOME/bin; \
echo "✔︎ $$HOME/bin created"; \
else \
echo "✔︎ $$HOME/bin already created"; \
fi
Thank you.
make itself performs expansion of $-prefixed characters; the $$ is expanded to a single literal $ to pass to the shell.
Consider a simple Makefile:
x=f
all:
xoo=3 && echo $xoo
which will output foo, because
make expands $x to the single character f.
make passes the string xoo=3 && echo foo to the shell for execution
Compare with
x=f
all:
xoo=3 && echo $$xoo
which outputs 3, because
make expands $$ to $
make passes the string xoo=3 && echo $xoo to the shell for execution

How to execute a file that is located in $PATH

I am trying to execute a hallo_word.sh that is stored at ~/bin from this script that is stored at my ~/Desktop. I have made both scripts executable. But all the time I get the problem message. Any ideas?
#!/bin/sh
clear
dir="$PATH"
read -p "which file you want to execute" fl
echo ""
for fl in $dir
do
if [ -x "$fl" ]
then
echo "executing=====>"
./$fl
else
echo "Problem"
fi
done
This line has two problems:
for fl in $dir
$PATH is colon separated, but for expects whitespace separated values. You can change that by setting the IFS variable. This changes the FIELD SEPARATOR used by tools like for and awk.
$fl contains the name of the file you want to execute, but you overwrite its value with the contents of $dir.
Fixed:
#!/bin/sh
clear
read -p "which file you want to execute" file
echo
IFS=:
for dir in $PATH ; do
if [ -x "$dir/$file" ]
then
echo "executing $dir/$file"
exec "$dir/$file"
fi
done
echo "Problem"
You could also be lazy and let a subshell handle it.
PATH=(whatever) bash command -v my_command
if [ $? -ne 0 ]; then
# Problem, could not be found.
else
# No problem
fi
There is no need to over-complicate things.
command(1) is a builtin command that allows you to check if a command exists.
The PATH value contains all the directories in which executable files can be run without explicit qualification. So you can just call the command directly.
#!/bin/sh
clear
# r for raw input, e to use readline, add a space for clarity
read -rep "Which file you want to execute? " fl || exit 1
echo ""
"$fl" || { echo "Problem" ; exit 1 ; }
I quote the name as it could have spaces.
To test if the command exists before execution use type -p
#!/bin/sh
clear
# r for raw input, e to use readline, add a space for clarity
read -rep "Which file you want to execute? " fl || exit 1
echo ""
type -p "$fq" >/dev/null || exit 1
"$fl" || { echo "Problem" ; exit 1 ; }

bash parameter expansion within a scalar variable via echo

title: bash parameter expansion within a scalar variable
I have a bash script which runs a diff between two files.
If there is a diff, I want it to print statement1 and statement2
They are long so i put them into variables, but the echo statement
will not expand the parameter.
Can this be done in bash?
#!/bin/bash
set -x
source="/home/casper"
target="data/scripts"
statement1="There is a change in ${i}, please check the file"
statement2="or cp /home/casper/${i} /data/scripts/$i"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff $source/$i $target/$i 2>&1 > /dev/null ; then
echo " "
else
echo "$statement1 "
echo "$statement2 "
fi
done
exit 0
The script seems to work - it finds a diff when it needs to find one.
However this is what it prints out.
There is a change in , please check the file
or cp /home/casper/ data/scripts/
I want it to say
There is a change in file2, please check the file
or cp /home/casper/file2 /data/scripts/file2
The problem is that $i is expanded when you define statement1 and statement2, not when you expand them. Use a shell function to output the text.
notification () {
echo "There is a change in $1, please check the file"
echo "or cp /home/casper/$1 /data/scripts/$1"
}
source="/home/casper"
target="data/scripts"
for i in file1 file2l file3 file4 file5 ; do
sleep 1 ;
if diff "$source/$i" "$target/$i" 2>&1 > /dev/null ; then
echo " "
else
notification "$i"
fi
done
exit 0
This can be done using eval:
TEMPLATE_MSG="aaa \${VALUE} ccc"
...
VALUE="bbb"
eval echo "${TEMPLATE_MSG}"
But I don't recommend it, because eval is evil :-) Other option is using pattern substitution:
TEMPLATE_MSG="aaa #1# ccc"
...
VALUE="bbb"
echo "${TEMPLATE_MSG/#1#/${VALUE}}"
So you put some unique pattern in your message (e.g. #1#) and then, when you print the message, you replace it with the content of variable.

Bash Script - Will not completely execute

I am writing a script that will take in 3 outputs and then search all files within a predefined path. However, my grep command seems to be breaking the script with error code 123. I have been staring at it for a while and cannot really seem the error so I was hoping someone could point out my error. Here is the code:
#! /bin/bash -e
#Check if path exists
if [ -z $ARCHIVE ]; then
echo "ARCHIVE NOT SET, PLEASE SET TO PROCEED."
echo "EXITING...."
exit 1
elif [ $# -ne 3 ]; then
echo "Illegal number of arguments"
echo "Please enter the date in yyyy mm dd"
echo "EXITING..."
exit 1
fi
filename=output.txt
#Simple signal handler
signal_handler()
{
echo ""
echo "Process killed or interrupted"
echo "Cleaning up files..."
rm -f out
echo "Finsihed"
exit 1
}
trap 'signal_handler' KILL
trap 'signal_handler' TERM
trap 'signal_handler' INT
echo "line 32"
echo $1 $2 $3
#Search for the TimeStamp field and replace the / and : characters
find $ARCHIVE | xargs grep -l "TimeStamp: $2/$3/$1"
echo "line 35"
fileSize=`wc -c out.txt | cut -f 1 -d ' '`
echo $fileSize
if [ $fileSize -ge 1 ]; then
echo "no"
xargs -n1 basename < $filename
else
echo "NO FILES EXIST"
fi
I added the echo's to know where it was breaking. My program prints out line 32 and the args but never line 35. When I check the exit code I get 123.
Thanks!
Notes:
ARCHIVE is set to a test directory, i.e. /home/'uname'/testDir
$1 $2 $3 == yyyy mm dd (ie a date)
In testDir there are N number of directories. Inside these directories there are data files that have contain data as well as a time tag. The time tag is of the following format: TimeStamp: 02/02/2004 at 20:38:01
The scripts goal is to find all files that have the date tag you are searching for.
Here's a simpler test case that demonstrates your problem:
#!/bin/bash -e
echo "This prints"
true | xargs false
echo "This does not"
The snippet exits with code 123.
The problem is that xargs exits with code 123 if any command fails. When xargs exits with non-zero status, -e causes the script to exit.
The quickest fix is to use || true to effectively ignore xargs' status:
#!/bin/bash -e
echo "This prints"
true | xargs false || true
echo "This now prints too"
The better fix is to not rely on -e, since this option is misleading and unpredictable.
xargs makes the error code 123 when grep returns a nonzero code even just once. Since you're using -e (#!/bin/bash -e), bash would exit the script when one of its commands return a nonzero exit code. Not using -e would allow your code to continue. Just disabling it on that part can be a solution too:
set +e ## Disable
find "$ARCHIVE" | xargs grep -l "TimeStamp: $2/$1/$3" ## If one of the files doesn't match the pattern, `grep` would return a nonzero code.
set -e ## Enable again.
Consider placing your variables around quotes to prevent word splitting as well like "$ARCHIVE".
-d '\n' may also be required if one of your files' filename contain spaces.
find "$ARCHIVE" | xargs -d '\n' grep -l "TimeStamp: $2/$1/$3"

Resources