gnu makefile: shell function execution order in a recipe - makefile

I'm using $(shell ...) gnu make function in a Makefile recipe, and it runs first before the preceding rows. Why?
A very simple example:
.PHONY: all
all:
#echo 1
#echo $(eval a=$(shell echo 2a 1>&2))2b
#echo 3 $(a)
The output is:
2a
1
2b
3
First runs the $(shell ...) line (2a), then the other lines.
How can I manage to run the $(shell ...) function when its row runs in the recipe, in this order?
1
2a
2b
3
Edit:
Without $(shell ...) it works as I expected:
.PHONY: all
all:
#echo 1
$(eval a=a)
#echo 2 $(a)
$(eval a=b)
#echo 3 $(a)
Output:
1
2 a
3 b
Edit 2:
Here is a part of the original Makefile. The pieces at >>> show the essence of my problem: I want to put the output of udisksctl into a make variable instead of file $#.loop (and do the same with $#.mount).
$(HDIMG): $(BOOTBLOCK_MBR_BIN) $(BOOTBLOCK_EXT2_BIN) $(LOADER_BIN) | $(DESTDIR)
dd if=/dev/zero of=$#.ext2 bs=1 seek=$(PSIZEB) count=0 2>/dev/null
$(MKFSEXT2) -F $#.ext2 >/dev/null
dd if=$(word 2,$^) of=$#.ext2 conv=notrunc 2>/dev/null
cp $< $#
dd if=/dev/zero of=$# bs=1 seek=$(HDSIZEB) count=0 2>/dev/null
echo $(PFDISK) | $(TR) | $(FDISK) $# >/dev/null
dd if=$#.ext2 of=$# bs=512 seek=$(PSTART) conv=sparse,notrunc iflag=fullblock 2>/dev/null
>>> udisksctl loop-setup --file $# --offset $(PSTARTB) --size $(PSIZEB) >$#.loop
sed -i -e 's/.* //;s/\.//' $#.loop
cat $#.loop
>>> udisksctl mount --block-device $$(cat $#.loop) >$#.mount
sed -i -e 's/.* //;s/\.//' $#.mount
cat $#.mount
#
mkdir -p $$(cat $#.mount)/boot/
cp $(word 2,$^) $$(cat $#.mount)/boot/
#/sbin/filefrag -b512 -e /
#
udisksctl unmount --block-device $$(cat $#.loop)
udisksctl loop-delete --block-device $$(cat $#.loop)
rm $#.loop

When make runs a recipe, it first expands all variables/functions in it, and then runs the shell commands line by line.
You can work around that by executing every shell command with $(shell ...), or, in your case, by using $(info ...) instead of echo:
.PHONY: all
all:
$(info 1)
$(info $(eval a=$(shell echo 2a 1>&2))2b)
$(info 3 $(a))
Output:
1
2a
2b
3
make: Nothing to be done for 'all'.
The Nothing to be done for 'all'. part is caused by the fact that after expanding the functions, the recipe is effectively empty (has 0 shell commands). Adding a no-op command (e.g. #true) to it removes the message.

Related

Using variables with export in Makefile

I would like to add some variables and echo commands in a Makefile. The section is:
.PHONY: run_harness
run_harness: link_dirs
#mkdir -p $(LOG_DIR)
#set -o pipefail && $(PYTHON3_CMD) code/main.py $(RUN_ARGS) --action="run_harness" 2>&1 | tee $(LOG_DIR)/stdout.txt
#set -o pipefail && $(PYTHON3_CMD) scripts/print_harness_result.py $(RUN_ARGS) 2>&1 | tee -a $(LOG_DIR)/stdout.txt
and I want to add
.PHONY: run_harness
run_harness: link_dirs
#mkdir -p $(LOG_DIR)
########
#export WRK="test"
#export startTime=$(date +%s)
#echo $(WRK), $(startTime)
########
#set -o pipefail && $(PYTHON3_CMD) code/main.py $(RUN_ARGS) --action="run_harness" 2>&1 | tee $(LOG_DIR)/stdout.txt
#set -o pipefail && $(PYTHON3_CMD) scripts/print_harness_result.py $(RUN_ARGS) 2>&1 | tee -a $(LOG_DIR)/stdout.txt
But those export and echo don't work because I get , in the output which mean both $(WRK) and $(startTime) are empty. How can I fix that?
UPDATE:
The following change
#mkdir -p $(LOG_DIR)
export WRK="3d-unet"; \
export startTime=$(date +%s); \
echo $(WRK), $(startTime) ;
doesn't work due to this error
Each line in a recipe is executed in a separate shell. You need to put all shell commands on one line, separated by semicolons (and possibly use line continuation with backslash newline for readability).
And you must reference shell variables with $$name (not with make variable $(name)). Note that make replaces a $$ with a single $ for the shell.
foo: bar
export x=hello y=world; \
echo "$$x" "$$y"

Missing Separator when using shell function?

I'm working on Solaris 11, fully patched. I'm trying to determine if the compiler supports an ISA by dumping preprocessor macros under the ISA.
Make is dying due to a Missing Separator. I'm having trouble finding information about the Missing Separator error when used with GNU make's shell function.
Here is the reduced case. There are no spaces, so its not a space/tab problem like in Make error: missing separator and friends.
$ cat -n GNUmakefile-test
1 EGREP ?= egrep
2 SUN_COMPILER := $(shell $(CXX) -V 2>&1 | $(EGREP) -i -c "CC: (Sun|Studio)")
3
4 # Begin SunCC
5 ifeq ($(SUN_COMPILER),1)
6 $(info "Sun compiler")
7 $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
8 ifeq ($(.SHELLSTATUS),0)
9 $(info "SSSE3")
10 SSSE3_FLAG = -xarch=ssse3 -D__SSSE3__=1
11 endif
12 endif
13 # End SunCC
14
15 all:
16 $(info "Do nothing")
The idea above is, SunCC does not provide macros for an ISA, like __AES__ or __SHA__. However, SunCC will error if the ISA is not supported, like -xarch=sha on SunCC 12.4. If I don't get an error, then I know the compiler supports the ISA, like -xarch=aes on SunCC 12.4. If there's an error I can fetch it from .SHELLSTATUS. (SunCC is not like Clang, GCC, Intel ICC, or MSVC in this area).
Here is the result:
$ CXX=/opt/solarisstudio12.4/bin/CC gmake -f GNUmakefile-test
"Sun compiler"
GNUmakefile-test:7: *** missing separator. Stop.
Where is the missing separator? Or, what is the real error that make is not reporting? Maybe something else?
My apologies for asking this question given how many times its been asked before.
I added tabs in an attempt to appease make. It produced the same error.
$ cat -n GNUmakefile-test
1 EGREP ?= egrep
2 SUN_COMPILER := $(shell $(CXX) -V 2>&1 | $(EGREP) -i -c "CC: (Sun|Studio)")
3
4 # Begin SunCC
5 ifeq ($(SUN_COMPILER),1)
6 $(info "Sun compiler")
7 $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
8 ifeq ($(.SHELLSTATUS),0)
9 $(info "SSSE3")
10 SSSE3_FLAG = -xarch=ssse3 -D__SSSE3__=1
11 endif
12 endif
13 # End SunCC
14
15 all:
16 $(info "Do nothing")
The way the shell function works (as discussed in the manual) is that it runs the command, and then it expands to the output of the command. That's why, when you see:
SUN_COMPILER := $(shell $(CXX) -V 2>&1 | $(EGREP) -i -c "CC: (Sun|Studio)")
the variable SUN_COMPILER is set to the output of that shell command.
So when you write this:
$(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
the command is run, then the output is substituted. After that, make tries to parse the results as make syntax. But the output of that command is clearly NOT make syntax, so you get this error.
If you don't care about the output and only care about the exit code, you need to either throw away the output:
$(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null >/dev/null 2>&1)
or else assign it to a dummy variable:
_x := $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
so that make assigns the result to a variable rather than thinking it's make syntax.
We decided to avoid .SHELLSTATUS since it seems to have some problems. We fell back to grep and the string "illegal value ignored".
ifeq ($(SUN_COMPILER),1)
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
SSSE3_FLAG = -xarch=ssse3 -D__SSSE3__=1
ARIA_FLAG = -xarch=ssse3 -D__SSSE3__=1
endif
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=sse4_2 -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
BLAKE2_FLAG = -xarch=sse4_2 -D__SSE4_2__=1
CRC_FLAG = -xarch=sse4_2 -D__SSE4_2__=1
endif
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=aes -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
GCM_FLAG = -xarch=aes -D__PCLMUL__=1
AES_FLAG = -xarch=aes -D__AES__=1
endif
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=sha -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
SHA_FLAG = -xarch=sha -D__SHA__=1
endif
endif
# End SunCC
We are not sure about -xarch=sha. At the moment we are trying to figure out what is needed.

Makefile - Select names from list using mask

I have Makefile in which I get a list of files from a certain directory. I need to choose only those, in the names of which there are numbers (like 123.txt). What is the best way to do it?
Thanks.
You can use a glob or wildcard, for example:
all.txt: text/*.txt
cat $^ >> $#
Usage:
$ cd -- "$(mktemp --directory)"
$ mkdir text
$ echo foo > text/1.txt
$ echo bar > text/2.txt
$ make
cat text/1.txt text/2.txt >> all.txt
NUMBER_FILENAMES := $(shell echo " $(LIST) " | sed 's/ [^ 0-9]* / /g')

Makefile with multiple values for a parameter

Currently, I am working on a makefile that takes a parameter "CLASS=xxx" and then compiles and does stuff with that value.
In the end, it runs an application ($APP) on a bunch of files.
I enter this command:
make default CLASS=Test_UART
and the makefile processes it thusly:
pc: $(APP)
make -C BUILDENV CLASS=$(CLASS) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes
../classCvt/classCvt <./Applications/$(CLASS).class> ./Applications/$(CLASS).ujc
time -p ./$(APP) ./Applications/$(CLASS).ujc `cat filelist.txt`
Hence it calls a makefile in my BUILDENV folder which does the following:
#USAGE: make -C <PATH_TO_THIS_FILES_PARENT_DIR> CLASS=<MY_JAVA_FILE_NAME_WITHOUT_JAVA_EXTENSION> OUT=<OUTPUT_FILE_NAME>
SELF := $(dir $(CURDIR)/$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))
CLASS ?= PLEASE_SPECIFY_CLASS_PARAM
DIR := $(PWD)#print working directory
CCVT ?= $(SELF)/../../classCvt/classCvt
TOBIN ?= $(SELF)/../../classCvt/tobin
OUTPUT ?=
### Comment: Defining CMD.
ifeq ($(BUILD), just_filelist)
CMD = echo
else
ifeq ($(BUILD), PC)
CMD = echo
else
ifeq ($(BUILD), unopt)
CMD = $(TOBIN)
else
### Comment: Optimized CMD = tobin -c ccvt
CMD = $(TOBIN) -c $(CCVT)
endif
endif
endif
ifeq ($(OUTPUT), )
OUT = &1
else
OUT = $(DIR)/$(OUTPUT)
endif
ifeq ($(SKIPSELF), yes)
MYCLASS =
else
MYCLASS = $(DIR)/Applications/$(CLASS).class
endif
all:
CLASSPATH=$(SELF)/RT/real:$(SELF)/RT/fake:$(DIR) javac $(DIR)/Applications/$(CLASS).java
find $(SELF)/RT/real -iname "*.class" -type f > $(SELF)/files
ls $(DIR)/Applications/*.class | grep -v "$(CLASS)\.class" >> $(SELF)/files || true
cat $(SELF)/files | xargs $(CMD) $(MYCLASS) >$(OUT)
rm -f $(SELF)/files
What I would like to do is give a command like:
make default CLASS=Test1,Test2,Test3
and the makefile to process it for the 3 classes and put the given classes in a .txt and the default classes in a different .txt, something like this like this:
pc: $(APP)
make -C BUILDENV default_classes BUILD=list_default_classes OUTPUT=list_default_classes.txt
# make -C BUILDENV given_classes BUILD=list_given_classes OUTPUT=list_given_classes.txt CLASS=$(CLASS) SKIPSELF=yes
../classCvt/classCvt `cat list_given_classes.txt`./Applications/$(CLASS).ujc
#here the list_given_classes should now contain the .ujc files
time -p ./$(APP) `cat list_given_classes.txt` `cat list_default_classes.txt`
and for the makefile in the BUILDENV, I expect something like:
default_classes:
CLASSPATH=$(SELF)/RT/real:$(SELF)/RT/fake:$(DIR)
find $(SELF)/RT/real -iname "*.class" -type f > $(SELF)/files
ls $(DIR)/Applications/*.class | grep -v "$(CLASS1)\.class" "$(CLASS2)\.class">> $(SELF)/files || true
cat $(SELF)/files | xargs $(CMD) >$(OUT)
rm -f $(SELF)/files
given_classes:
javac $(DIR)/Applications/$(CLASS).java
find $(SELF)/RT/real -iname "*.class" -type f > $(SELF)/files
ls $(DIR)/Applications/*.class | grep -v "$(CLASS)\.class" >> $(SELF)/files || true
cat $(SELF)/files | xargs $(CMD) $(MYCLASS) >$(OUT)
rm -f $(SELF)/files
However, I'm not sure how to do this for a CLASS parameter containing multiple classes.
I'm thinking to try and parse the Test1,Test2,Test3 value into a list of 1,2,3 and then iterating over it. But no clue if this is a good way and even on how to do it.
What do you guys suggest?
Pretty way:
pc: $(APP)
define BUILD_CLASS
pc: pc-$(CLASS_SPLIT)
pc-$(CLASS_SPLIT):
make -C BUILDENV CLASS=$(CLASS_SPLIT) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes
../classCvt/classCvt <./Applications/$(CLASS_SPLIT).class> ./Applications/$(CLASS_SPLIT).ujc
time -p ./$(APP) ./Applications/$(CLASS_SPLIT).ujc `cat filelist.txt`
endef
CLASSES := $(shell echo $(CLASS) | tr ',' ' ')
$(foreach CLASS_SPLIT, $(CLASSES), $(eval $(BUILD_CLASS)))
Simple way:
pc: $(APP)
$(foreach C, $(shell echo $(CLASS) | tr ',' ' '), \
make -C BUILDENV CLASS=$(C) BUILD=just_filelist OUTPUT=filelist.txt SKIPSELF=yes && \
../classCvt/classCvt <./Applications/$(C).class> ./Applications/$(C).ujc && \
time -p ./$(APP) ./Applications/$(C).ujc `cat filelist.txt` &&) true

Compiling a program in a loop with qsub

I want to compile a program in a bash for loop. When I run the program from the command line it will compile but when I use qsub it doesn't compile.
Is there something I am missing?
Regards,
John
Bash File
#!/bin/bash
#$ -N runTest
#$ -m e
#$ -r y
cd /afs/crc.nd.edu/user/private/NDPICMCC/SAFECODE
thisDir="SAFECODE"
t=2000
originalGUILine="pres = 7.6E1"
oldGUILine="$originalGUILine"
GUIfile="GUIVars.f90"
for (( i = -3 ; i <= 1 ; i=i+1 ))
do
p="7.6E$i"
newGUILine="pres = $p"
sed -i "s/$oldGUILine/$newGUILine/g" "$GUIfile"
oldGUILine="$newGUILine"
make clean >& /dev/null
make 1D >& /dev/null
make 1D
./PressurePIC
cp "Anode_ele_eng.csv" "../results/T_${t}_P_${p}_Energies.csv"
done
sed -i "s/$oldGUILine/$originalGUILine/g" "$GUIfile"
makefile
CC=ifort
OPTIONS = -warn noalign -autodouble
PRNG = luxury.f90
MAIN = NDPIC1D.v0.f90 GUIVars.f90 GlobalVars.f90 StatisticalDistribution.f90 Emission.f90
TODO = ParticleInCell.f90 MonteCarloCollision.f90 Transformations.f90
EXE = PressurePIC
all:
#$(CC) $(PRNG) $(MAIN) $(TODO) -o $(EXE) $(OPTIONS)
1D:
#$(CC) $(PRNG) $(MAIN) $(TODO) -o $(EXE) $(OPTIONS)
2D:
#$(CC) $(PRNG) $(MAIN) $(TODO) -o $(EXE) $(OPTIONS)
clean:
#rm *.mod $(EXE)
run:
#./$(EXE)
info:
#echo $(EXE)
I'm not sure what flavour of qsub you are using so this may not help ...
Try including
#$ -V
at the start of your script. On some systems which process qsub it ensures that environment variables are exported from your environment to the environment in which the script eventually runs. These environments are not, generally, the same.

Resources