how to build packages in Parallel to reduce the time taken by make command(Golang)
my current directory is "thor"
inside my Makefile
build:
go build -o thor
go build thor/package1
go build thor/scripts/package2
in the above case ,all the packages are stand alone processes which run independently . when i run make command each packages build one by one . so if each package build take 30 sec ,totally it takes 90 sec(30 sec * 3) . but if i can build these packages in parallel ,it will only take 30 sec.
in case of shell scripts this kind of cases can be handled by running each scripts in background with & and wait till the end of each scripts using wait
sample code
#!/bin/bash
echo "Starts"
./s1.sh &
./s2.sh &
./s3.sh &
wait
echo "ends"
In the above cases all the scripts s1.sh ,s2.sh ,s3.sh will run concurrently and wait till all the process finished.
So , can we do something like this in Makefile also :)
Instead of
build:
go build -o thor
go build thor/package1
go build thor/scripts/package2
split this in three separate recipes, something like this
build: build1 build2 build3
build1:
go build -o thor
build2:
go build thor/package1
build3:
go build thor/scripts/package2
.PHONY: build build1 build2 build3
Then call make with the -j option and you're done.
It would be better to have targets that correspond to files that are actually created by your go build command, together with a list of prerequisites. Then you don't need .PHONY and make can decide whether a rebuild is actually needed.
You can use gox that will parallelize builds for multiple platforms.
By default, Gox will parallelize based on the number of CPUs you have and build for every platform by default
Exactly the same as go build, build using
gox
If you want to build packages and sub-packages:
gox ./...
Related
I have makefile all: clean $(binary_output_path)/$(target_executable) which cleans output directory first and the build executable. The problem is when I want to use -j10: sometimes it start building and cleaning at the same time, so build fails, obviously.
How can I overcome this and have targets executed in order but on multiple cores?
You should add the dependency:
$(binary_output_path)/$(target_executable): | clean
However, this is not a good idea to use clean like that at all. Just have
all: $(binary_output_path)/$(target_executable)
$(binary_output_path)/$(target_executable): prerequisites
write your recipe here that builds this target
You should focus on writing makefiles in such a way that cleaning is not needed. If you must clean first before building your executable, find out why that is so, and write your commands that build the executable, in such a way that they work regardless of whether you have "cleaned" or not.
I'm running a project using Go Modules with 1.11.4 on Ubuntu, running in WSL.
My problem is that I'm having getting incremental builds to work as I expect. Perhaps this is due to me misunderstanding how it's supposed to, but I'd be glad if someone could clarify if that is the case.
Just as an example, if I do go build ./... then everything is built, as expected.
If I now do go build ./... again without any changes, my expectation was that due to the incremental builds, this time nothing would be built. But it builds everything again. I tried doing go build -i ./... (even though my understanding is that -i isn't needed anymore from 1.10), but the result is the same. This has been puzzling me for some time, as after reading the documentation I indeed expected the go build command to produce incremental builds.
The other day I realized that if instead I do go install ./... first, and then go install ./... again a second time, the second time around nothing is built, as I would expect. If I change a single module and run go install ./... again, then only that module is rebuilt, again what I would expect. So this gives me incremental builds.
So my questions are
1) Did I misunderstand go build ./... and how it handles incremental builds? Do I need to use go install instead?
2) Typically, we build the modules one by one, using the -o flag to specify an output path. Using go install instead, there is no -o option to specify an output path. Is there anything I can do to achieve something similar to -o using go install?
Thanks!
I have a makefile that has a ton of subsystem and am able to build it with the -j flag so that it goes much faster and builds the different recipes in parallel.
This seems to be working fine for now but am not sure if I am missing some needed dependencies and am just "getting lucky" with the order that these are being built.
Is there a way where I can randomize the order recipes are run while still following all the dependencies that I have defined?
You can control number of jobs Make is allowed to run asynchronously with -j command line option. This way you can "randomize" recipes being executed simultaneously and catch some issues in your makefiles.
I'll duplicate the answer from https://stackoverflow.com/a/72722756/5610270 here:
Next release of GNU make will have --shuffle mode. It will allow
you to execute prerequisites in random order to shake out missing
dependencies by running $ make --shuffle.
The feature was recently added in
https://savannah.gnu.org/bugs/index.php?62100 and so far is available
only in GNU make's git tree.
I am pretty new to Rust and I want to time the execution of my program. I searched online, but found nothing so far. After running cargo build, my code is executed like this:
bling#bling ~/github/rust/hello_world [master *]
± % ./target/debug/hello_world
Does Cargo have a built in way to time the execution or do I need to use the command line?
I don't think you can pass a flag to cargo directly to time the compilation step. The simplest way to do it is to use the time command line utility:
$ time ./target/debug/hello_world
./target/debug/hello_world 3.02s user 0.18s system 34% cpu 9.177 total
Cargo does have something similar: cargo bench allows you to write benchmarks for your program, although it is only available on nightly Rust. This gives you very specific reports about the speed of certain parts of your program. The docs have more reading.
$ cargo bench
running 1 test
test bench_xor_1000_ints ... bench: 131 ns/iter (+/- 3)
Cargo does not have a built in way to time things. You will want to use your operating systems mechanism. For example, on Linux or macOS, you can probably use time (I use hyperfine myself):
time ./target/debug/hello_world
Of special note is that you have a debug build. This has no optimizations and should not be used for profiling. Instead, you should build your code in release mode:
cargo build --release
and then execute the program in the target/release directory.
Also, you probably do not want to include the time for Cargo itself. That is, do not do either of these:
time cargo run
time cargo run --release
It's not baked into Cargo, but hyperfine is written in Rust and is an alternative to time:
cargo install hyperfine
hyperfine ./target/debug/hello_world
I have a makefile that is a 3rdParty dependency builder, so it's actually just going to various other directories and running cmake/make with various flags to ensure all 15-20 dependencies of my project compile the way I need.
Building parallel would really help here, (the build takes about 2 hours serially), but I need a 'make -jN' to not run the toplevel makefile parallel, instead run it serially (the various 3rdParty libs have internal dependencies to meet) and pass the arg to the inside makefiles.
Is there a way to get this behavior?
Use the .NOTPARALLEL pseudo target; from the docs:
`.NOTPARALLEL'
If `.NOTPARALLEL' is mentioned as a target, then this invocation of
`make' will be run serially, even if the `-j' option is given.
Any recursively invoked `make' command will still be run in
parallel (unless its makefile contains this target). Any
prerequisites on this target are ignored.