Suppose you have a simple makefile which consists of a pattern rule such as:
all: 1_end.fa 2_end.fa 3_end.fa
%_end.fa: %_start.fa
program $^ > $#
When you run such a makefile with nohup, the STDOUT from each process is dumped into the same nohup.out file. Is it possible to create a rule where the STDOUT from each process is dumped into its own file-specific nohup.out file?
Update 1:
I have managed to work this out, but I'm still open for better suggestions. My method uses a sister makefile to run the pipeline makefile, passing a command line variable to the processes and renaming the nohup on running. Such as:
pipeline.GNUmakefile:
all: $(source)_end.fa
$(source)_end.fa: $(source)_start.fa
program $^ > $'
config.GNUmakefile:
sources = $(wildcard *.fa)
nohups = $(sources.fa=.out)
all: $(nohups)
%.out: %.fa
nohup make -f pipeline.GNUmakefile source=$(basename $^) > $# &
For each source, create a temporary directory, cd there, run nohup from there, rename the output file from there to whatever you want, delete the temporary directory.
Really has nothing to do with makefiles.
Related
Thanks for all your time and response -
Currently, we are using the nested build, multiple Makefiles, and individual subdirectories having their own Makefile, all are connected with a top-level Makefile. We are running
make xxxxx_yyyy_defconfig
make
this builds and creates an output file which is xxxxx.elf file. --- Till here everything works fine.
Now we are having multiple def-configs(around 50), I want to build all configurations using one "make all" command. is that possible?
This is not a simple case where we can put all "all: prog01 prog02 prog03" as every program needs to have a different configuration. Configuration can be achieved by using "make xxxxx_yyyy_defconfig". The output of "make config" is the .config file, which is used during the "make" command.
Based on .config file many variables are exported which is used at the subdirectory level.
So How can I build multiple configurations using a single "make all" command?
Environment - Ubuntu, Cross compile for ARM, output file xxxx.elf.
With the use of script and make file I am able to solve, But I have to solve only using Makefile.
in Makefile add one PHONY target
all:
./build_all.sh #shell script calling.
Created one shell script like this
#! /usr/bin/bash
echo "Make All"
for entry in `ls conf`; do
make $entry
wait
make
if [ $? -eq 0 ] ; then
for xxxfile in `ls xxx*_*` ; do
xxxdir=$(echo $xxxfile | cut -b yy-zz)
mkdir -p $xxxdir
mv $xxxfile $xxxdir/
done
else
break
fi
done
If you want to build several configurations you must do this out of tree in separate build directories (make O=/tmp/builds/foo foo_defconfig; make -C /tmp/builds/foo) to avoid conflicts. A shell script could do this as well as a Makefile but if you insist on using a Makefile you could try the following that assumes your source tree is in /src/kernel and you want to build configuration foo in /tmp/builds/foo; adapt to your needs:
$ pwd
/tmp/builds
$ cat Makefile
CONFIGS := uuuu_vvvv xxxx_yyyy ...
BUILD := /tmp/build
KERNEL := /src/kernel
.PHONY: $(CONFIGS) all
all: $(CONFIGS)
$(CONFIGS):
rm -rf $#
mkdir -p $(BUILD)/$#
$(MAKE) -C $(KERNEL) O=$(BUILD)/$# $#_defconfig
$(MAKE) -C $(BUILD)/$#
$ make
I have the following folder structure
1st-grade-math-class/
common/
mystyle.sty
mysubstyle.sty
fonts/
font1.ttf
font2.ttf
font3.ttf
week01/
handout.tex
image1.pdf
image2.pdf
week02/
handout.tex
image1.pdf
image2.pdf
...
week13/
handout.tex
output/
[empty]
And I would like to create a Makefile - in the best way - to do the following:
make sure I include the common directory properly in TEXINPUTS
compile the handout.tex into a PDF (using either pdflatex or xelatex) and have it in the output directory as week01-handout-student.pdf
compile the handout.tex with a line of LaTeX prepended to the beginning of the file (that sets a flag) into a PDF and have it in the output directory as week01-handout-teacher.pdf
clean everything up (the log, aux, etc. files)
I am not sure I know how to this in any other way than manually duplicating an elementary Makefile/bash script in every subdirectory, then calling each of them one by one with a for loop.
I would appreciate help on how to build this process, ideally with a single Makefile in the root directory. Thanks.
UPDATE: I purposefully did not want to give any details about how I compile LaTeX, in case somebody has a better suggestion than my current usage. Right now I am using Latexmk (which is already a make-like wrapper of LaTeX):
latexmk -pdf file.tex generates file.pdf
to add the line of code, I do a simple echo "line of code" > temp.tex and cat handout.tex >> temp.tex, then the same latexmk command
latexmk -c file.tex in a directory cleans all temporary files used to compile file.tex
TEXINPUTS is the TeX path variable, to let TeX find (in its path) the style files: I do TEXINPUTS=full-path-to/common and then export TEXINPUTS before compiling anything.
If anybody has a better suggestion, I am a willing taker.
Something like this should do what you want I believe:
OUTDIR := output
# Tell make to export this environment variable to all recipe lines it runs.
export TEXINPUTS := $(abspath common)
# Get the list of all of our week directories.
weekdirs := $(wildcard week*)
#$(info weekdirs:$(weekdirs))
# Create student output filenames from week directory names.
STUDENT_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-student.pdf,$(weekdirs))
#$(info STUDENT_HANDOUTS:$(STUDENT_HANDOUTS))
# Create teacher output filenames from week directory names.
TEACHER_HANDOUTS := $(patsubst %,$(OUTDIR)/%-handout-teacher.pdf,$(weekdirs))
#$(info TEACHER_HANDOUTS:$(TEACHER_HANDOUTS))
# Default target depends on all output files.
all: $(STUDENT_HANDOUTS) $(TEACHER_HANDOUTS)
# Pattern rule for building pdf files.
%.pdf:
#echo + Making $# from $^
#echo cd $(#D) && echo latexmx -pdf $(abspath $<)
#echo cd $(#D) && echo latexmk -c $(abspath $<)
# Static pattern rule mapping student output files to input files.
$(STUDENT_HANDOUTS) : $(OUTDIR)/%-handout-student.pdf : %/handout.tex
# Pattern rule to generate temporary input files from original input files.
%/handout-tmp.tex: %/handout.tex
#echo echo 'line of code' '>' $#
#echo cat $^ '>>' $#
# Static pattern rule mapping teacher output files to (temporary) input files.
$(TEACHER_HANDOUTS) : $(OUTDIR)/%-handout-teacher.pdf : %/handout-tmp.tex
Uncomment the $(info) lines to see a bit of how the variables are put together.
This uses latexmk -c to clean up auxiliary files after creating the output files.
Technique 1. The echo foo >$#; cat $< >>$# thing is something I've done before; I think it's reasonable.
Technique 2. Another technique is to have your .tex document, or a package file it uses, include a line like:
\InputIfFileExists{lecturenotes.config}{}{}
That allows you to make certain adjustments in a makefile, as in for example:
# any bits of configuration that should be in all the .config files
SWITCHES=\makeindex
%-notes.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{sidenotes}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
%-single.pdf: %.tex
printf '$(SWITCHES)\\ExecuteOptions{oneside}\n' >lecturenotes.config
TEXINPUTS=styles: pdflatex $<
mv ${<:.tex=.pdf} $#
rm lecturenotes.config
Technique 3. A third technique for controlling LaTeX from outside is to include in your document (or in a package file):
\newif\ifdemonstrator
\expandafter\ifx\csname demonstrator\endcsname\relax
\demonstratorfalse
\else
\demonstratortrue
\fi
...
\ifdemonstrator
\typeout{Demonstrator is TRUE}
\else
\typeout{Demonstrator is FALSE}
\fi
Then call latex with:
%-plain.pdf: %.tex
latex $<
mv ${<:.tex=.pdf} $#
%-demo.pdf: %.tex
latex --jobname ${<:.tex=} '\def\demonstrator{x}\input{$<}`
mv ${<:.tex=.pdf} $#
Technique 1 is a bit of a blunt instrument, but if that's all that's needed, it's pretty lightweight.
Technique 2 is probably the most neatly engineered, but it is slightly more effort.
Technique 3 is probably the one I use most in this sort of circumstance.
I'm trying to pipe to the output of the compiler to the tee command in windows, but I've ran into an issue where if the compiler fails within make it'll continue compiling the next file when I want to it to stop. Is there a way to have the exit status of the first command be the exit status of the second command?
$(ODIR)/%.o: %.c $(DEPS)
$(CC) -c -o $# $< $(CFLAGS) 2>&1 | tee build_log.txt
First off, I would leave any logging to the caller of make. Second off, this sort of piping is untidy in make. Third off, not a fan of losing the stderr stream inside make.
That said, this is a shell question. If you are using bash then see pipefail in the manual. Unfortunately I think it's quite tricky to turn on. (Yeah, I know you said windows, but I assume you aren't using the execrable cmd.)
SHELL := /bin/bash
passes:
(exit 1) |& cat
fails:
bash -c 'set -o pipefail; (exit 1) |& cat'
After Struggling a lot, I came to this solution ...
.ONESHELL:
$(ODIR)/%.o: %.c $(DEPS)
$(CC) -c -o $# $< $(CFLAGS) 2> temp_err_file
set EXIT_STATUS=%ERRORLEVEL%
type temp_err_file >> build_log.txt
type temp_err_file 1>&2
del /q temp_err_file
exit /b %EXIT_STATUS%
Here .ONESHELL allows make to run entire recipe in the single shell command instead of running each line in separate cmd and collect return status of each separately. Overall exit status depends on the main compilation command so in end it is necessary to exit with the status of compilation.
I know its not a clean solution involving the temp_err_file and if something goes wrong after compilation command, make would not be able to catch it but this I think is the best I can find to work with windows without losing stderr stream and logging.
A method that is shell independent and may be feasible in some cases is the following:
assume you have a recipe:
target:
try_making_target |& tee target.log
What I did was convert it to:
target:
(try_making_target || rm -f $#) |& tee target.log
test -e $#
The the piped command fails, the "fallback" (command after ||) will delete the goal file, and the final test will fail. Note that this example assumes the OS is Linux ('rm' for deletions) and that your shell supports the || operator.
This assumes that you are not interested in partial results when try_making_target fails. If you want to keep partial result in 'target', you can use other "marker" files to designate the success or failure of try_making_target. Something like this may work:
target:
touch $#.succeeded # Assume success
(try_making_target || rm -f $#.succeeded) |& tee target.log # Delete to mark failure
test -e $#.succeeded # Fail if marker
rm $#.succeeded # Remove unneeded marker
This last code would also work for phony goals, although you really should minimize the use of these (I try only using them as mnemonics for real files, which may have longish names).
This is what I mean:
ls -l *.ec
For each result found, I would like to compile it one at a time.
e.g., for $something.ec:
esql $something.ec -o $something
$something.ec is original file
$something is target exe file.
My question is: how can I put that into a loop to go through all the files in my directory?
You can use a for-loop as follows:
for file in *.ec
do
esql "$file" -o "${file//.ec}"
done
I recommend you write a simple Makefile:
INPUTS = $(wildcard *.ec)
all: $(INPUTS:.ec=)
%: %.ec
esql $# -o $<
and then simply run make (or make -B to force execution) whenever you want to re-execute them.
I'm trying to modify a makefile to cross-compile binaries. The command in question is below:
# GNU Make solution makefile autogenerated by Premake
# Type "make help" for usage help
ifndef config
config=debug
endif
export config
PROJECTS := json openjaus openjaus-core openjaus-environment openjaus-mobility openjaus-manipulator openjaus-ugv Base Managed PingTest LargeMessageTest PdDemo GposDemo GposClientDemo StillImageSensorDemo StillImageClientDemo
.PHONY: all clean help $(PROJECTS)
all: $(PROJECTS)
json:
#echo "==== Building json ($(config)) ===="
#${MAKE} --no-print-directory -C .build -f json.make
As can be seen the makefile has several targets. They all have the same structure as the 'json' target. The command in question in question is
#${MAKE} --no-print-directory -C .build -f json.make
The '${MAKE}' variable = make (I have verified this with echo)
What does the -C do?
What does the .build do?
I'm good with -f json.make
Also, when I run make the json.make file gets created compiles file and deletes it self, so I do not have access to that file.
The error I receive when I modify the command in question is
==== Building json (debug) ====
make[1]: Nothing to be done for `/home/botbear/openwrt/trunk/staging_dir/toolchain- arm_v6k_gcc-linaro_uClibc-0.9.32_eabi/bin/arm-openwrt-linux-c++'.
The command after modifications looks like:
#${MAKE} /home/botbear/openwrt/trunk/staging_dir/toolchain-arm_v6k_gcc-linaro_uClibc-0.9.32_eabi/bin/arm-openwrt-linux-c++ --no-print-directory -C .build -f json.make
Any help is appreciated!
you can use man make to understand the parameters for make:
-C dir, --directory=dir
Change to directory dir before reading the makefiles or doing any-
thing else. If multiple -C options are specified, each is inter-
preted relative to the previous one: -C / -C etc is equivalent to
-C /etc. This is typically used with recursive invocations of
make.
-f file, --file=file, --makefile=FILE
Use file as a makefile.
so -C .build changes to the directory .build.
and i don't understand the part of your question about modifying the command.
Try to find where json.make lives. It seems that it's that makefile which creates & deletes the directory you were talking about.
From the command line it seems that make changes directory to .build and executes the json.make. Not sure how json.make ends up there. Is .build the directory which is created and then deleted?