Compiling a program in a loop with qsub - bash

I want to compile a program in a bash for loop. When I run the program from the command line it will compile but when I use qsub it doesn't compile.
Is there something I am missing?
Regards,
John
Bash File
#!/bin/bash
#$ -N runTest
#$ -m e
#$ -r y
cd /afs/crc.nd.edu/user/private/NDPICMCC/SAFECODE
thisDir="SAFECODE"
t=2000
originalGUILine="pres = 7.6E1"
oldGUILine="$originalGUILine"
GUIfile="GUIVars.f90"
for (( i = -3 ; i <= 1 ; i=i+1 ))
do
p="7.6E$i"
newGUILine="pres = $p"
sed -i "s/$oldGUILine/$newGUILine/g" "$GUIfile"
oldGUILine="$newGUILine"
make clean >& /dev/null
make 1D >& /dev/null
make 1D
./PressurePIC
cp "Anode_ele_eng.csv" "../results/T_${t}_P_${p}_Energies.csv"
done
sed -i "s/$oldGUILine/$originalGUILine/g" "$GUIfile"
makefile
CC=ifort
OPTIONS = -warn noalign -autodouble
PRNG = luxury.f90
MAIN = NDPIC1D.v0.f90 GUIVars.f90 GlobalVars.f90 StatisticalDistribution.f90 Emission.f90
TODO = ParticleInCell.f90 MonteCarloCollision.f90 Transformations.f90
EXE = PressurePIC
all:
#$(CC) $(PRNG) $(MAIN) $(TODO) -o $(EXE) $(OPTIONS)
1D:
#$(CC) $(PRNG) $(MAIN) $(TODO) -o $(EXE) $(OPTIONS)
2D:
#$(CC) $(PRNG) $(MAIN) $(TODO) -o $(EXE) $(OPTIONS)
clean:
#rm *.mod $(EXE)
run:
#./$(EXE)
info:
#echo $(EXE)

I'm not sure what flavour of qsub you are using so this may not help ...
Try including
#$ -V
at the start of your script. On some systems which process qsub it ensures that environment variables are exported from your environment to the environment in which the script eventually runs. These environments are not, generally, the same.

Related

makefile missing separator showing up when using modified makefile for macOS

I am trying to use the makefile for my work related to SPheno. I am on macOS. When I try to use the command "make Model=Scotogenic". it's showing "*** missing separator". I went through lot of forums and added tabs. when I tried to do it again , now it's showing "*** commands commence before first target" . I am using the default textedit in macOS. I am attaching both the code and screenshot. The terminal screenshot
Code:
# PreDef = -DGENERATIONMIXING -DONLYDOUBLE -DSEESAWIII
PreDef = -DGENERATIONMIXING -DONLYDOUBLE
# setting various paths
InDir = ../include
Mdir = ${InDir}
name = ../lib/libSPheno.a
#
# options for various compilers
#
# Intels ifort, default in optimized mode
F90 = ifort
comp = -c -O -module ${Mdir} -I${InDir}
LFlagsB = -O
# Intels ifort, debug modus
ifeq (${F90},ifortg)
F90 = ifort
comp = -c -g -module ${Mdir} -I${InDir}
LFlagsB = -g
endif
gfortran
ifeq (${F90},gfortran)
comp = -c -O -J${Mdir} -I${InDir}
LFlagsB = -O
endif
# g95
ifeq (${F90},g95)
comp = -c -O -fmod=${Mdir} -I${InDir}
LFlagsB = -O
endif
# Lahey F95 compiler
ifeq (${F90},lf95)
comp = -c -O -M ${Mdir} -I${InDir}
LFlagsB = -O
endif
# NAG f95/2003
ifeq (${F90},nagfor)
comp = -c -O -DONLYDOUBLE -mdir ${Mdir} -I${InDir}
LFlagsB = -O
endif
.SUFFIXES : .o .ps .f90 .F90 .a
bin/SPheno: ${name} SPheno4.o
${F90} -o SPheno ${LFlagsB} SPheno4.o ../lib/${name}
mv SPheno ../bin
${name}: ${name}(Control.o) ${name}(Mathematics.o) ${name}(RGEs.o) \
${name}(MathematicsQP.o) ${name}(LoopFunctions.o) ${name}(StandardModel.o) \
${name}(Model_Data.o) ${name}(Couplings.o) ${name}(SusyMasses.o) \
${name}(LoopCouplings.o) ${name}(DecayFunctions.o) \
${name}(SusyDecays.o) ${name}(ThreeBodyPhaseSpace.o) \
${name}(ThreeBodyPhaseSpaceS.o) ${name}(Chargino3.o) \
${name}(Gluino3.o) ${name}(Neutralino3.o) \
${name}(Stop3BodyDecays.o) ${name}(Slepton3Body.o) ${name}(BranchingRatios.o) \
${name}(EplusEminusProduction.o) ${name}(TwoLoopHiggsMass.o) \
${name}(LoopMasses.o) ${name}(SugraRuns.o) ${name}(Experiment.o) \
${name}(LowEnergy.o) ${name}(NMSSM_tools.o) ${name}(RPtools.o) \
${name}(LHC_observables.o) ${name}(InputOutput.o)
clean:
rm -f *.o *~ */*.o */*~
cleanall:
rm -f bin/SPheno4 lib/*.a *~ */*.o */*~ include/*
#
# Suffix rules
#
.f90.a:
${F90} ${comp} $<
ar -ruc $# $*.o
rm -f $*.o
.F90.a:
${F90} ${comp} ${PreDef} $<
ar -ruc $# $*.o
rm -f $*.o
.f90.o:
${F90} ${comp} $<
.f90.ps:
a2ps -o $*.ps $<
.h.ps:
a2ps -o $*.ps $<

Makefile eval function causing unexpected end of file

I'm trying to increment a Makefile variable with eval function inside a rule (to do a fancy loading bar), by doing
$(eval COUNTER=$(shell echo $$(($(COUNTER) + 1))))
I found on internet that this is working for some people but this line causes me a Syntax Error: unexpected end of file, what should I do to increment my counter? Are there some non-horrific alternatives to this function, or a way to avoid this unexpected end of file? thanks for your help
Here is the entire Makefile I'm working on
SHELL = /bin/sh
.SUFFIXES = .c .h .o
NAME = Necklace
SRCD = src
INCD = inc
OBJD = .obj
SRC = $(addsuffix $(word 1, $(.SUFFIXES)),\
main\
necklace\
utils)
INC = $(addsuffix $(word 2, $(.SUFFIXES)),\
necklace)
OBJ = $(SRC:$(word 1, $(.SUFFIXES))=$(word 3, $(.SUFFIXES)))
CC = gcc
CFLAGS = -Wall\
-Werror\
-Wextra\
-I $(INCD)
COUNT = $(shell cat file.count 2>/dev/null)
COMPILED = $(shell echo $$(($(shell ls src/*$(word 1, $(.SUFFIXES)) | wc -l | tr -d ' ') - $(COUNT))))
define set
set = $(eval $1 := $2)
ifeq ($(COUNT),)
all:
echo $(words $(SRC)) > file.count && $(MAKE) -n | grep $(CC) | wc -l | tr -d ' ' > tmp.txt && rm -f file.count && echo $$(($$(cat tmp.txt) - 1)) > file.count && make
else
all: $(NAME)
endif
$(NAME): $(addprefix $(OBJD)/, $(OBJ)) | $(addprefix $(INCD)/, $(INC))
$(CC) $(CFLAGS) $^ -o $#
$(OBJD):
mkdir $#
$(OBJD)/%.o: $(SRCD)/%.c | $(OBJD)
#printf '\r\033[2KCompiling [' && sh prc.sh $(COMPILED) $(COUNT) && printf ']' && $(CC) $(CFLAGS) -c $< -o $# && $(eval COMPILED:=$(shell echo $$(($(COMPILED) + 1)))) #Syntax Error is here
clean:
rm -rf $(OBJD)
fclean: clean
rm -rf $(NAME)
re: fclean all
I managed to do it by writing and reading the variable into a file instead of using Make variables

Missing Separator when using shell function?

I'm working on Solaris 11, fully patched. I'm trying to determine if the compiler supports an ISA by dumping preprocessor macros under the ISA.
Make is dying due to a Missing Separator. I'm having trouble finding information about the Missing Separator error when used with GNU make's shell function.
Here is the reduced case. There are no spaces, so its not a space/tab problem like in Make error: missing separator and friends.
$ cat -n GNUmakefile-test
1 EGREP ?= egrep
2 SUN_COMPILER := $(shell $(CXX) -V 2>&1 | $(EGREP) -i -c "CC: (Sun|Studio)")
3
4 # Begin SunCC
5 ifeq ($(SUN_COMPILER),1)
6 $(info "Sun compiler")
7 $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
8 ifeq ($(.SHELLSTATUS),0)
9 $(info "SSSE3")
10 SSSE3_FLAG = -xarch=ssse3 -D__SSSE3__=1
11 endif
12 endif
13 # End SunCC
14
15 all:
16 $(info "Do nothing")
The idea above is, SunCC does not provide macros for an ISA, like __AES__ or __SHA__. However, SunCC will error if the ISA is not supported, like -xarch=sha on SunCC 12.4. If I don't get an error, then I know the compiler supports the ISA, like -xarch=aes on SunCC 12.4. If there's an error I can fetch it from .SHELLSTATUS. (SunCC is not like Clang, GCC, Intel ICC, or MSVC in this area).
Here is the result:
$ CXX=/opt/solarisstudio12.4/bin/CC gmake -f GNUmakefile-test
"Sun compiler"
GNUmakefile-test:7: *** missing separator. Stop.
Where is the missing separator? Or, what is the real error that make is not reporting? Maybe something else?
My apologies for asking this question given how many times its been asked before.
I added tabs in an attempt to appease make. It produced the same error.
$ cat -n GNUmakefile-test
1 EGREP ?= egrep
2 SUN_COMPILER := $(shell $(CXX) -V 2>&1 | $(EGREP) -i -c "CC: (Sun|Studio)")
3
4 # Begin SunCC
5 ifeq ($(SUN_COMPILER),1)
6 $(info "Sun compiler")
7 $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
8 ifeq ($(.SHELLSTATUS),0)
9 $(info "SSSE3")
10 SSSE3_FLAG = -xarch=ssse3 -D__SSSE3__=1
11 endif
12 endif
13 # End SunCC
14
15 all:
16 $(info "Do nothing")
The way the shell function works (as discussed in the manual) is that it runs the command, and then it expands to the output of the command. That's why, when you see:
SUN_COMPILER := $(shell $(CXX) -V 2>&1 | $(EGREP) -i -c "CC: (Sun|Studio)")
the variable SUN_COMPILER is set to the output of that shell command.
So when you write this:
$(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
the command is run, then the output is substituted. After that, make tries to parse the results as make syntax. But the output of that command is clearly NOT make syntax, so you get this error.
If you don't care about the output and only care about the exit code, you need to either throw away the output:
$(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null >/dev/null 2>&1)
or else assign it to a dummy variable:
_x := $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>/dev/null)
so that make assigns the result to a variable rather than thinking it's make syntax.
We decided to avoid .SHELLSTATUS since it seems to have some problems. We fell back to grep and the string "illegal value ignored".
ifeq ($(SUN_COMPILER),1)
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=ssse3 -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
SSSE3_FLAG = -xarch=ssse3 -D__SSSE3__=1
ARIA_FLAG = -xarch=ssse3 -D__SSSE3__=1
endif
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=sse4_2 -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
BLAKE2_FLAG = -xarch=sse4_2 -D__SSE4_2__=1
CRC_FLAG = -xarch=sse4_2 -D__SSE4_2__=1
endif
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=aes -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
GCM_FLAG = -xarch=aes -D__PCLMUL__=1
AES_FLAG = -xarch=aes -D__AES__=1
endif
COUNT := $(shell $(CXX) $(CXXFLAGS) -E -xarch=sha -xdumpmacros /dev/null 2>&1 | $(EGREP) -i -c "illegal value ignored")
ifeq ($(COUNT),0)
SHA_FLAG = -xarch=sha -D__SHA__=1
endif
endif
# End SunCC
We are not sure about -xarch=sha. At the moment we are trying to figure out what is needed.

Makefile to compile lists of source files

I have lists of files that I want my Makefile to compile, one list for each source language:
CFILES= Src/Application/main.c Src/Core/data.c Lib/routines.c
ASFILES= Src/Application/startup.s Lib/sqrt.s
I want all the output in one directory:
OBJDIR= output
How do I do the equivalent of:
output/main.o : Src/Application/main.c
cc -c -o output/main.o Src/Application/main.c
output/data.o : Src/Core/data.c
cc -c -o output/data.o Src/Core/data.c
output/routines.o : Lib/routines.c
cc -c -o output/routines.o Lib/routines.c
output/startup.o : Src/Application/startup.s
as -o output/startup.o Src/Application/startup.s
output/sqrt.o : Lib/sqrt.s
as -o output/sqrt.o Lib/sqrt.s
The recipes are the same for every file in its list.
The input files are in all sorts of different directories and it is not acceptable to just list their filenames and use a search path to find them, their explicit paths must be used.
The output filename is the basename of the source file name with the extension changed to o. There are no duplicated basenames between the lists for the different source languages.
I do not want to have to list the object files, this should be derived from the source lists.
I am using gnu make, but bonus points for a portable solution.
Something like the following could do:
all :
OBJDIR := output
CFILES := Src/Application/main.c Src/Core/data.c Lib/routines.c
ASFILES := Src/Application/startup.s Lib/sqrt.s
target = ${OBJDIR}/$(patsubst %.s,%.o,$(notdir ${1}))
obj.c :=
obj.s :=
define obj
$(call target,${1}) : ${1} | ${OBJDIR}
obj$(suffix ${1}) += $(call target,${1})
${1} : ; mkdir -p `dirname $$#` && touch $$# # Create the source for testing. Remove this.
endef
define SOURCES
$(foreach src,${1},$(eval $(call obj,${src})))
endef
$(eval $(call SOURCES,${CFILES}))
$(eval $(call SOURCES,${ASFILES}))
all : ${obj.c} ${obj.s}
${obj.c} : % :
#echo cc -c -o $# $^; touch $# # echo and touch are for testing. Remove these.
${obj.s} : % :
#echo as -o $# $^; touch $# # echo and touch are for testing. Remove these.
${OBJDIR} :
mkdir $#
.PHONY: all
Output:
$ make
make: Entering directory '/home/max/tmp'
mkdir -p `dirname Src/Application/main.c` && touch Src/Application/main.c # Create the source for testing. Remove this.
mkdir output
cc -c -o output/main.c Src/Application/main.c
mkdir -p `dirname Src/Core/data.c` && touch Src/Core/data.c # Create the source for testing. Remove this.
cc -c -o output/data.c Src/Core/data.c
mkdir -p `dirname Lib/routines.c` && touch Lib/routines.c # Create the source for testing. Remove this.
cc -c -o output/routines.c Lib/routines.c
mkdir -p `dirname Src/Application/startup.s` && touch Src/Application/startup.s # Create the source for testing. Remove this.
as -o output/startup.o Src/Application/startup.s
mkdir -p `dirname Lib/sqrt.s` && touch Lib/sqrt.s # Create the source for testing. Remove this.
as -o output/sqrt.o Lib/sqrt.s
make: Leaving directory '/home/max/tmp'
I have little experience in writing makefiles, so this is just an attempt. In my example I have C and C++ files in a few directories and build a program made of these files.
$ cat Makefile
.PHONY : clean all
CC=gcc
CXX=g++
CFILES = c/f.c c/g.c
CPPFILES = cpp/main.cpp
OUTPUT = ./output
SOURCE_DIRS := $(dir $(CFILES))
SOURCE_DIRS += $(dir $(CPPFILES))
VPATH = $(sort $(SOURCE_DIRS))
C_FILENAMES := $(notdir $(CFILES))
CPP_FILENAMES += $(notdir $(CPPFILES))
OBJ_FILES := $(patsubst %.c, $(OUTPUT)/%.o, $(C_FILENAMES) )
OBJ_FILES += $(patsubst %.cpp, $(OUTPUT)/%.o, $(CPP_FILENAMES) )
all : $(OUTPUT)/program
$(OUTPUT)/program : $(OBJ_FILES)
g++ -o $# $^
$(OUTPUT)/%.o : %.cpp
$(shell mkdir -p $(OUTPUT) )
$(CXX) $(CXXFLAGS) -c $< -o $#
$(OUTPUT)/%.o : %.c
$(shell mkdir -p $(OUTPUT) )
$(CC) $(CFLAGS) -c $< -o $#
clean:
rm -fr $(OUTPUT)
And this is an example of using my makefile:
$ make
gcc -c c/f.c -o output/f.o
gcc -c c/g.c -o output/g.o
g++ -c cpp/main.cpp -o output/main.o
g++ -o output/program output/f.o output/g.o output/main.o
The following method should do what you want: compile all of your specified source files and put the object files in the directory ./output automatically. Of course, you need to provide compiler options, proper libraries necessary for linking, and so on.
OBJDIR =./output
SRCDIR1 =./Src/Application
SRCDIR2 =./Src/Core
SRCDIR3 =./Lib
SRC1 =$(SRCDIR1)/main.c
SRC2 =$(SRCDIR2)/data.c
SRC3 =$(SRCDIR3)/routines.c
SRC4 =$(SRCDIR1)/startup.s
SRC5 =$(SRCDIR3)/sqrt.s
OBJ1 =$(patsubst $(SRCDIR1)/%.c,$(OBJDIR)/%.o,$(SRC1))
OBJ2 =$(patsubst $(SRCDIR2)/%.c,$(OBJDIR)/%.o,$(SRC2))
OBJ3 =$(patsubst $(SRCDIR3)/%.c,$(OBJDIR)/%.o,$(SRC3))
OBJ4 =$(patsubst $(SRCDIR1)/%.s,$(OBJDIR)/%.o,$(SRC4))
OBJ5 =$(patsubst $(SRCDIR3)/%.s,$(OBJDIR)/%.o,$(SRC5))
vpath %.c $(SRCDIR1): $(SRCDIR2): $(SRCDIR3)
vpath %.s $(SRCDIR1): $(SRCDIR3)
all: $(OBJ1) $(OBJ2) $(OBJ3) $(OBJ4) $(OBJ5)
cc $^ -o executable
$(OBJDIR)/%.o: %.c | $(OBJDIR)
cc -c $< -o $#
$(OBJDIR)/%.o: %.s | $(OBJDIR)
cc -c $< -o $#
$(OBJDIR):
mkdir -p $(OBJDIR)

Insert dependency in parallel makefile

I'm trying to add an extra dependency to a rule in a parallel makefile. I might have found a way, but I'm in doubt. (I haven't written the original makefile and I'm not an expert in make.)
The original makefile looks like this:
VER = busybox-1.16.2
URL = http://busybox.net/downloads/$(VER).tar.bz2
export KBUILD_OUTPUT = $(ROOTDIR)/user/busybox/build-$(VER)
all: build-$(VER)/.config depmod.pl
$(MAKE) -C build-$(VER)
build-$(VER)/.config: $(ROOTDIR)/config/.config
mkdir -p build-$(VER)
sed -n \
-e '/_CROSS_COMPILER_PREFIX=/s:=.*:="$(CROSS_COMPILE)":' \
-e '/CONFIG_USER_BUSYBOX_/s:CONFIG_USER_BUSYBOX_:CONFIG_:p' \
$< > $#.uclinux-dist.new
set -e ; \
if [ ! -e $# ] || ! cmp -s $#.uclinux-dist.new $#.uclinux-dist.old ; then \
cp $#.uclinux-dist.new $#.uclinux-dist.old ; \
cp $#.uclinux-dist.old $# ; \
yes "" | $(MAKE) -C $(VER) oldconfig ; \
fi
depmod.pl: $(VER)/examples/depmod.pl
ln -sf $< $#
I want to add a 'download' rule to the make.
$(VER)/: $(VER).tar.bz2
tar -jxvf $(VER).tar.bz2
touch $#
$(VER).tar.bz2:
wget $(URL)
touch $#
This rule must be executed before anything else. The parallel build prevent constructs like
all: |$(VER)/ build-$(VER)/.config depmod.pl
(This works in single threaded builds.)
My solution so far is this:
VER = busybox-1.18.5
URL = http://busybox.net/downloads/$(VER).tar.bz2
export KBUILD_OUTPUT = $(ROOTDIR)/user/busybox/build-$(VER)
all: build-$(VER)/.config depmod.pl
$(MAKE) -C build-$(VER)
$(VER)/: $(VER).tar.bz2
tar -jxvf $(VER).tar.bz2
touch $#
$(VER).tar.bz2:
wget $(URL)
touch $#
build-$(VER)/.config: $(ROOTDIR)/config/.config | $(VER)/
mkdir -p build-$(VER)
sed -n \
-e '/_CROSS_COMPILER_PREFIX=/s:=.*:="$(CROSS_COMPILE)":' \
-e '/CONFIG_USER_BUSYBOX_/s:CONFIG_USER_BUSYBOX_:CONFIG_:p' \
$< > $#.uclinux-dist.new
set -e ; \
if [ ! -e $# ] || ! cmp -s $#.uclinux-dist.new $#.uclinux-dist.old ; then \
cp $#.uclinux-dist.new $#.uclinux-dist.old ; \
cp $#.uclinux-dist.old $# ; \
yes "" | $(MAKE) -C $(VER) oldconfig ; \
fi
depmod.pl: $(VER)/examples/depmod.pl
ln -sf $< $#
$(VER)/examples/depmod.pl: | $(VER)/
Problem is, I don't really know what kind of magic the depmod.pl rule is. Is it executed correctly, now that I've added an explicit empty rule?
I'd hate to answer my own question, but I think I've found the answer.
The dependency of the depmod.pl rule is not real/relevant in the original script.
The code:
depmod.pl: $(VER)/examples/depmod.pl
ln -sf $< $#
Ought to be written:
depmod.pl:
ln -sf $(VER)/examples/depmod.pl $#
So my extra empty rule makes no difference. In the new script, the dependency almost makes sence though.

Resources