I am learning Makefile and trying to implement parallelism. I am aware of the "-j" option. However, for example having the following makefile (on Windows)-
all: a b c d
a:
# some build rule
b:
# some build rule with parallelism
c:
# some build rule
d:
#some build rule
I am trying to run make all with only target "b" running in parallel. Passing the -j option with the build rule for "b" doesn't work. Any pointers?
You could get b's recipe to run in the background as so:
all: a b c d
#echo running $#
.PHONY: a b c d all
a c d: | b
#echo -n _$#0 && \
sleep 1 && echo -n _$#1 && \
sleep 1 && echo _$#2
b:
#(echo -n _$#0 && \
sleep 2 && echo -n _$#1 && \
sleep 2 && echo -n _$#2\
) &
Which outputs:
_b0_a0_a1_b1_a2
_c0_c1_b2_c2
_d0_d1_d2
running all
The order-only dependency on b makes b run first, otherwise it wouldn't start until after a completes with -j1... It does of course mean that you have to build b if you build either a c or d.
Alternatively, (and I'm not recommending this) you could use some manual locking mechanism such as flock to prevent a, c, and d from running in parallel (note that the flock only protects a single shell, so you would have to collapse your recipes into a single line protected by flock for this to work).
Related
I have the following Makefile target:
target1:
$(eval count_abc := $(shell grep -c "ABC" myFileA))
$(eval count_def := $(shell grep -c "DEF" myFileB))
echo $(count_abc)
echo $(count_def)
ifeq ($(count_abc),$(count_def))
echo "TRUE"
else
echo "FALSE"
endif
But the output is always TRUE, e.g.:
echo 22
22
echo 21
21
echo TRUE
TRUE
What am I doing wrong here? What I want is INSIDE the target do 2 greps and compare their outputs and do something or something else based on the result. Please note that the greps must be done within the target since myFileA and myFileB get created on the target before and don't exist at the beginning when running make.
Thanks,
Amir
The rule file for "make" is declarative in nature - the makefile defines rules and targets, and then the make program evaluate the rules, and decide which action to take based on the target. As a result, execution is not always in the order the lines are entered into the file.
More specifically, the "ifeq" is evaluated at the rule definition stage, but the actions for building the target (eval count_abc ...) are executed when the target is built. As a result, when the ifeq is processed, both count_abc and count_def are still uninitialized, expanded to empty strings.
For the specific case you described - building a target that will compare the grep -c output from the two files, you can try something like below, effectively using shell variables (evaluated when target is evaluated), and not make variables (which are mostly declarative, evaluated when makefile is read)
target1:
count_abc=$(grep -c "ABC" myFileA) ; \
count_def=$(grep -c "DEF" myFileB) ; \
echo $(count_abc) ; \
echo $(count_def) ; \
if [ "$count_abc" -eq "$count_def" ] ; then echo TRUE ; else echo FALSE ; fi
Disclaimer: I did not run the revised makefile, not having access to desktop at this time.
I am trying to open and externally managed shell and trying to run some command in makefile. How can I do that?
For example, if I want to run the following sequence in my make file:
> python
> a = 6
> b = 5
> c = a + b
> print(c)
>exit()
This is a sample of externally managed shell that I am trying.
There are various ways, none of them super-simple. Here's the most basic one:
runcommand:
( echo 'a = 6'; \
echo 'b = 5'; \
echo 'c = a + b'; \
echo 'print(c)'; \
echo 'exit()'; \
) | python
Basically, if you can write the command in a single shell command line, then you can transpose that command line into your makefile recipe.
I have a very large and convoluted make system that takes hours to complete. I'm researching potential optimizations, and I had a question on whether it is possible to prioritize certain targets.
Here's a sample of a simple Makefile that illustrates the concept:
Test%: Prog%
#run some post build verification on Prog{n}
# -- takes 30min or less on failure...
Prog1: A B
# do some linking
Prog2: B C
# do some linking
A B C:
# take 10 minutes each
And then I run
make -j2 Prog1 Test2
Then the fastest way to build this would be: B & C, followed by Prog2 & A, followed by Test2 and Prog2. Of course, typically, make would build A and B first, which would delay my final output by 10 minutes.
If I have foreknowledge that Test2 is going to be a bottleneck, is it possible to prioritize that target and all its dependencies over other targets in gnu make?
To add to the question, Let's say I'm building Test1, Test2 ... Test20. Then in that case, I'd like one of the targets -- say Test1 to be completed as soon as possible, as if it that fails, I would like to know as soon as possible so I can terminate the build and let someone else use the build machine.
I considered doing separate instances of make as so:
make Test2 && make Prog1
But, while the test is building, then Prog1 will not be building, resulting in wasted cycles on the build server. If I tried to build them in parrallel with priorities:
`make Test2; nice -n -10 make Prog1`
This could result in race conditions when building B.
I've not yet found any good solutions, but I thought I'd ask on in-case I missed something. (Also, I'm not sure if this should be on SO or SuperUser -- I chose here as it's technically about 'programming', but please feel free to correct me and I can move it).
GNU make offers no syntax to add a weight to a target. I.e. when make is able to start the next job and the independent(!) targets A and B are unblocked (all their dependencies have been fulfilled) and require remaking, then it depends on their order in the internal make database which one gets selected first for execution.
But you could use additional dependencies to achieve your goal. To take the example from your question:
TESTS := Test1 Test2
$(TESTS): Test%: Prog%
Prog1: A B
Prog2: B C
ifdef _PRIORITIZE_TEST2
# additional dependencies to make sure Prog2 -> Test2 is prioritized
A: B C
endif
A B C Prog1 Prog2 Test1 Test2 all:
#echo $#
.PHONY: A B C Prog1 Prog2 $(TESTS) all
Test runs:
$ make --trace -j10 Prog1 Test2
Makefile:13: target 'A' does not exist
echo A
Makefile:13: target 'B' does not exist
echo B
Makefile:13: target 'C' does not exist
echo C
A
B
C
Makefile:13: update target 'Prog1' due to: A B
echo Prog1
Makefile:13: update target 'Prog2' due to: B C
echo Prog2
Prog1
Prog2
Makefile:13: update target 'Test2' due to: Prog2
echo Test2
Test2
$ make --trace -j10 _PRIORITIZE_TEST2=1 Prog1 Test2
Makefile:13: target 'B' does not exist
echo B
Makefile:13: target 'C' does not exist
echo C
B
C
Makefile:13: update target 'A' due to: B C
echo A
Makefile:13: update target 'Prog2' due to: B C
echo Prog2
A
Makefile:13: update target 'Prog1' due to: A B
echo Prog1
Prog2
Makefile:13: update target 'Test2' due to: Prog2
echo Test2
Prog1
Test2
This will probably involve a lot of hand-crafting to get the result you want. You might even need to write different sets of additional dependencies to cover different make invocation scenarios.
You could look into dumping the GNU make database (make --no-builtin-rules --print-data-base), parsing it to extract all targets & their dependencies and visualizing the resulting graph. Here is an example how to generate a directed graph in the DOT language of Graphviz:
#!/usr/bin/perl
use warnings;
use strict;
BEGIN {
print "digraph build\n";
print "{\n";
}
my $default_goal = '???';
my #goals;
while (<STDIN>) {
if (my($target, $dependencies) = /^([\w_-]+):\s+(.*)/) {
#print "Node ${target}\n";
print " $_ -> ${target}\n"
for (split(' ', $dependencies));
} elsif (my($goals) = /^MAKECMDGOALS :=\s+(.+)/) {
#goals = split(' ', $goals);
} elsif (my($goal) = /^\.DEFAULT_GOAL :=\s+(.+)/) {
$default_goal = $goal;
}
}
END {
#goals = ( $default_goal )
unless (#goals);
#print "Root $_\n"
# for (#goals);
print "}\n";
}
exit 0;
For the above example that would result in:
$ make -n --no-builtin-rules --print-data-base Prog1 Test2 2>&1 | perl dummy.pl
digraph build
{
A -> Prog1
B -> Prog1
Prog1 -> Test1
B -> Prog2
C -> Prog2
Prog2 -> Test2
}
I got confused with Makefile. I am trying to run a simple command in the Makefile but it gives me the error "/bin/bash: line 3: :=: command not found". I am using shell to run this makefile
This is my part of my Makefile:
all:
vlog Benchmarks/$(NAME)/Syn/*.v
$(eval tux_number := 1)
$(eval range := 1)
$(eval ssh_log := 255)
echo "Start Range: ${range}"
echo "tux-number: ${tux_number}"
while [[ $$range -le 50 ]] ; do \
ssh -l yazdanbakhsh tux-$(tux_number).cae.wisc.edu exit ; \
echo "range: ${range}" ; \
eval $$range := $$((${range}+1)) ; \
done
Thanks
all:
#range=1; \
while [ $$range -le 10 ] ; \
do echo Range: $$range; \
let range=range+1 ; \
done;
Note that the whitespace in front of #range... is the only TAB.
Just to fix your obvious problems with Makefile syntax, here is an attempt at refactoring your attempt into valid code.
tux_number := 1
ssh_log := 255 # not used anywhere
all:
vlog Benchmarks/$(NAME)/Syn/*.v
echo "Start Range: 1" # This is probably no longer very useful output
echo "tux-number: ${tux_number}"
range=1; while [ $$range -le 50 ] ; do \
ssh -l yazdanbakhsh tux-$(tux_number).cae.wisc.edu exit ; \
echo "range: $$range" ; \
range=$$(expr "$$range + 1); \
done
Notice how tux_number and ssh_log are Makefile variables, while range only exists in the shell which executes the while loop. I have avoided the Bashisms in order to make this portable. (If portability is not important, you might want to refactor it back to Bash syntax and use for ((range=1; range<=50; range++)); do... instead.)
Your use of eval is misguided. As you can see, I simply lifted out the Makefile variables outside the recipe where they don't belong. What you were doing was (1) have Make evaluate the expression range := 1 (which evaluates to itself) and (2) use the output as a shell command in a recipe. Since it's not a valid shell command, you got the syntax error from Bash. Without further ado, I'll just take the easy way out here and say that eval is a complex subject, and until you get more experience with Make, it's probably just best to forget that it exists.
In order to properly make use of Make's facilities, I would make this parallelizable, i.e. split it up into 50 individual targets. This is a bit clumsy (there's probably a better way to define range here), but at least it should illustrate a number of differences to your approach. (If you don't insist on having range count up from 1, making it zero-based would make this a little less clumsy. This exploits the fact that the empty string is harmless in a shell snippet, so we can use it instead of a zero prefix. Again, this could be simplifed if you don't care about the human readability of the range index.)
digits := 0 1 2 3 4 5 6 7 8 9
deca := "" 1 2 3 4
range := $(filter-out ""0,$(foreach d,$(deca),$(foreach i,$(digits),$d$i))) 50
# Or, at the expense of an external process,
# range := $(shell perl -le 'print $$_ for 1..50')
.PHONY: all
all: $(patsubst %,ssh-%,$(range))
.PHONY: ssh-%
ssh-%:
ssh -l yazdanbakhsh tux-$(tux_number).cae.wisc.edu exit
echo "range: $*"
This can be run with something like make -j 5 to execute these in parallel batches of five, for example.
Incidentally, the commented-out $(shell ...) call might be the actual answer to your question, if what you really wanted to do was to use Make to drive an external program to calculate something for you.
a:
#echo 1
b:
#echo 2
c:
#if [ ! -d somefolder ]; \
then \
# how do I invoke target b here?
fi;
How can I invoke target b inside target c, based on my condition? Sort of like an antcall, if you're familiar with ant.
Just say make b