After reading answer to this question:
Make "make" default to "make -j 8"
I am wondering if there is way to make the -j option automatically use the correct number of compile threads?
So I say make. And the make command itself uses 6 or 4 or 8 threads depending on the hardware?
make does not look up the number of cores by itself if you just use make -j -- instead, it parallelizes to the max. However, you should be able to determine the number of cores by
grep -c "^processor" /proc/cpuinfo
or (as per Azor-Ahai's comment, if available on your system)
nproc
Hence:
make -j $(nproc)
See "How How to obtain the number of CPUs/cores in Linux from the command line?" for more details. Also, see GNU make: should the number of jobs equal the number of CPU cores in a system?
Related
Sorry if this is a repeat question and I know there are a lot of similar questions out there, but I am really struggling to find a simple answer that works.
I want to run an executable many times eg.
seq 100 | xargs -Iz ./program
However I would like to run this over multiple cores on my machine (currently on a macbook pro so 4 cores) to speed things up.
I have tried using gnu parallel by looking at other answers on here as that seems to be what I want and have it installed but I can't work out how parallel works and what arguments I need in what order. Non of the readme is helping as it is trying to do much more complicated things than I want to.
Could anyone help me?
Thanks
So, in order to run ./program 100 times, with GNU Parallel all you need is:
parallel -N0 ./program ::: {1..100}
If your CPU has 8 cores, it will keep 8 running in parallel till all jobs are done. If you want to run, say 12, in parallel:
parallel -j 12 -N0 ./program ::: {1..100}
Let's say I have N Fortran executables and M cores on my machine, where N is greater than M. I want to be able to run these executables in parallel. I am using RHEL 6.9
I have used both OpenMP and GNU Parallel in the past to run code in parallel. However for my current purposes, neither of these two options would work: RHEL doesn't have a GNU Parallel distribution, and OpenMP applies to parallelizing blocks within a single executable, not multiple executables.
What is the best way to run these N executables in parallel? Would a simple approach like
executable_1 & executable_2 & ... & executable_N
work?
Just because it is not part of the official repository, doesn't mean you cannot use GNU parallel on a RHEL system. Just build GNU parallel yourself or install a third party rpm.
xargs supports parallel execution as well. Its interface is not ideal for your use case, but this should work:
echo executable_1 executable_2 ... executable_N | xargs -n1 -P8 bash -c
(-P8 means “run eight processes in parallel”.)
For more complex tasks, I sometimes write makefiles and use make -j8 to run targets in parallel.
Here I read
If no value is provided for the number of copies to execute (i.e.,
neither the "-np" nor its synonyms are provided on the command line),
Open MPI will automatically execute a copy of the program on each
process slot (see below for description of a "process slot")
So I would expect
mpirun program
to run eight copies of the program (actually a simple hello world), since I have an Intel® Core™ i7-2630QM CPU # 2.00GHz × 8, but it doesn't: it simply runs a single process.
If you do not specify the number of processes to be used, mpirun tries to obtain them from the (specified or) default host file. From the corresponding section of the man page you linked:
If the hostfile does not provide slots information, a default of 1 is assumed.
Since you did not modify this file (I assume), mpirun will use one slot only.
On my machine, the default host file is located in
/etc/openmpi-x86_64/openmpi-default-hostfile
i7-2630QM is a 4-core CPU with two hardware threads per core. With computationally intensive programs, you should better start four MPI processes instead of eight.
Simply use mpiexec -n 4 ... as you do not need a hostfile for starting processes on the same node where mpiexec is executed.
Hostfiles are used when launching MPI processes on remote nodes. If you really need to create one, the following should do it:
hostname slots=4 max_slots=8
(replace hostname with the host name of the machine)
Run the program as
mpiexec -hostfile name_of_hostfile ...
max_slots=8 allows you to oversubscribe the node with up to eight MPI processes if your MPI program can make use of the hyperthreading. You can also set the environment variable OMPI_MCA_orte_default_hostfile to the full path of the hostfile instead of explicitly passing it each and every time as a parameter to mpiexec.
If you happen to be using a distributed resource manager like Torque, LSF, SGE, etc., then, if properly compiled, Open MPI integrates with the environment and builds a host and slot list from the reservation automatically.
From the man page for gnu make:
The ‘-j’ option is a special case (see Parallel Execution). If you set
it to some numeric value ‘N’ and your operating system supports it
(most any UNIX system will; others typically won’t), the parent make
and all the sub-makes will communicate to ensure that there are only
‘N’ jobs running at the same time between them all. Note that any job
that is marked recursive (see Instead of Executing Recipes) doesn’t
count against the total jobs (otherwise we could get ‘N’ sub-makes
running and have no slots left over for any real work!)
If your operating system doesn’t support the above communication, then
‘-j 1’ is always put into MAKEFLAGS instead of the value you
specified. This is because if the ‘-j’ option were passed down to
sub-makes, you would get many more jobs running in parallel than you
asked for. If you give ‘-j’ with no numeric argument, meaning to run
as many jobs as possible in parallel, this is passed down, since
multiple infinities are no more than one.
Which common operating systems support or don't support this behavior?
And how can you tell if your os supports it?
To tell if your make supports this, run this command from your shell prompt:
echo 'all:;#echo $(filter jobserver,$(.FEATURES))' | make -f-
If it prints 'jobserver', then you have support. If it prints nothing, you do not have support. Or, if your OS doesn't support echo or pipelines, create a small makefile containing:
all:;#echo $(filter jobserver,$(.FEATURES))
then run make with that makefile.
A frequent problem I encounter is having to run some script with 50 or so different parameterizations. In the old days, I'd write something like (e.g.)
for i in `seq 1 50`
do
./myscript $i
done
In the modern era though, all my machines can handle 4 or 8 threads at once. The scripts aren't multithreaded, so what I want to be able to do is run 4 or 8 parameterizations at a time, and to automatically start new jobs as the old ones finish. I can rig up a haphazard system myself (and have in the past), but I suspect that there must be a linux utility that does this already. Any suggestions?
GNU parallel does this. With it, your example becomes:
parallel ./myscript -- `seq 1 50`