Here I read
If no value is provided for the number of copies to execute (i.e.,
neither the "-np" nor its synonyms are provided on the command line),
Open MPI will automatically execute a copy of the program on each
process slot (see below for description of a "process slot")
So I would expect
mpirun program
to run eight copies of the program (actually a simple hello world), since I have an Intel® Core™ i7-2630QM CPU # 2.00GHz × 8, but it doesn't: it simply runs a single process.
If you do not specify the number of processes to be used, mpirun tries to obtain them from the (specified or) default host file. From the corresponding section of the man page you linked:
If the hostfile does not provide slots information, a default of 1 is assumed.
Since you did not modify this file (I assume), mpirun will use one slot only.
On my machine, the default host file is located in
/etc/openmpi-x86_64/openmpi-default-hostfile
i7-2630QM is a 4-core CPU with two hardware threads per core. With computationally intensive programs, you should better start four MPI processes instead of eight.
Simply use mpiexec -n 4 ... as you do not need a hostfile for starting processes on the same node where mpiexec is executed.
Hostfiles are used when launching MPI processes on remote nodes. If you really need to create one, the following should do it:
hostname slots=4 max_slots=8
(replace hostname with the host name of the machine)
Run the program as
mpiexec -hostfile name_of_hostfile ...
max_slots=8 allows you to oversubscribe the node with up to eight MPI processes if your MPI program can make use of the hyperthreading. You can also set the environment variable OMPI_MCA_orte_default_hostfile to the full path of the hostfile instead of explicitly passing it each and every time as a parameter to mpiexec.
If you happen to be using a distributed resource manager like Torque, LSF, SGE, etc., then, if properly compiled, Open MPI integrates with the environment and builds a host and slot list from the reservation automatically.
Related
I have a code that runs on 2 MPI process, each of them using multiple threads with OpenMP. My computer (Ubuntu) has 4 CPU (cores ?). Thus I made a small script to give 2 CPU per process such that each of them can run in parallel:
# Environment variables
export CPU_PER_PROC=2
export OMP_NUM_THREADS=${CPU_PER_PROC}
export OPTION="-map-by node:PE=${CPU_PER_PROC}"
mpiexec ${OPTION} -n 2 python3 main.py
But now I would like to give, for instance, 3 CPU to the process (rank) 0 and only 1 CPU to the rank 1 such that:
Process 0 : use 3 CPU with X (certainly 3 for better performances) threads.
Process 1 : use 1 CPU with Y (certainly 1) threads.
All MPI process inherit from the parent environment by default which contains the OMP_NUM_THREADS variable. To tweak the value regarding the rank, one solution is to change the environment variable in the child processes. In Python, you can do that using for example:
import os
os.environ['OMP_NUM_THREADS'] = str(3)
This line must be execute before the OpenMP initialization (that is before the first execution of a parallel directive). You can tweak the right-end-side regarding the MPI rank.
An alternative solution is to use the omp_set_num_threads function call but this is not so simple from a Python code. Another solution is to use a num_thread clause in a parallel section, but again, this is generally an issue from a Python code. A last alternative is to calla shell script that changes the environment before calling the Python script but this script can hardly get the MPI rank (AFAIK there are shell variable to get this but they are dependent of the MPI implementation and so not portable).
Given: 2 Ubuntu 16.04 machines with multiple CPU cores.
I want to execute multiple instances of program fixed_arg arg2on the machines, passing one file name per call as arg2 to the program.
So far, working with xargs, this works on a single machine:
find . -iname "*.ext" -print | xargs -n1 -P12 program fixed_arg
(This will find all files with extension "ext" in the current directory (.), print one file per line (-print), and call xargs to call program 12 times in parallel (-P12) with only one argument arg2per call (-n1). Note the white space on the end of the whole command.)
I want to use multiple machines on which I installed the "mpich" package from the official Ubuntu 16.04 repositories.
I just do not know how to make mpiexec to run my program with only one argument on multiple machines.
I do know that mpiexec will accept a list of arguments, but my list will be in the range of 800 to 2000 files which so far has been to long for any program.
Any help is appreciated.
You just selected wrong instrument (Or give us more details about your target program). MPI (mpich implementation, mpiexec and mpirun commands) is not for starting unrelated programs on multiple hosts, it is for starting one program with exactly same source code in the way, when program knows now many copies are there (up to 100 and more thousands) to do well-defined point-to-point and collective message passing between copies. It is instrument to parallel some scientific codes like computation over huge array which can't computed on single machine or even can't fit into its memory.
Better instrument for you can be GNU parallel (https://www.gnu.org/software/parallel/); and if you have one or two machines or it is just several runs, it is easier to manually split your file list in two parts, and run two parallel or xargs on every machine (by hand or with ssh using authorized_keys). I'll assume that all files are accessible from both machines at the same path (NFS share or something like; no magic tool like mpi or gnu parallel will forward files for you, but batch some modern batch processing system may):
find . -iname "*.ext" -print > list
l=$(wc -l < list)
sp=$((l/2))
split -l $sp list
cat xaa | xargs -n1 -P12 program fixed_arg &
cat xab | ssh SECOND_HOST xargs -n1 -P12 program fixed_arg &
wait
Or just learn about multi-host usage of GNU parallel: https://www.gnu.org/software/parallel/man.html
-S #hostgroup Distribute jobs to remote computers. The jobs will be run on a list of remote computers. GNU parallel will determine the number of CPU cores on the remote computers and run the number of jobs as specified by -j.
EXAMPLE: Using remote computers
It also has a magic of sending files to remote machine with --transferfile filename option if you have no shared FS between two ubuntus.
From the man page for gnu make:
The ‘-j’ option is a special case (see Parallel Execution). If you set
it to some numeric value ‘N’ and your operating system supports it
(most any UNIX system will; others typically won’t), the parent make
and all the sub-makes will communicate to ensure that there are only
‘N’ jobs running at the same time between them all. Note that any job
that is marked recursive (see Instead of Executing Recipes) doesn’t
count against the total jobs (otherwise we could get ‘N’ sub-makes
running and have no slots left over for any real work!)
If your operating system doesn’t support the above communication, then
‘-j 1’ is always put into MAKEFLAGS instead of the value you
specified. This is because if the ‘-j’ option were passed down to
sub-makes, you would get many more jobs running in parallel than you
asked for. If you give ‘-j’ with no numeric argument, meaning to run
as many jobs as possible in parallel, this is passed down, since
multiple infinities are no more than one.
Which common operating systems support or don't support this behavior?
And how can you tell if your os supports it?
To tell if your make supports this, run this command from your shell prompt:
echo 'all:;#echo $(filter jobserver,$(.FEATURES))' | make -f-
If it prints 'jobserver', then you have support. If it prints nothing, you do not have support. Or, if your OS doesn't support echo or pipelines, create a small makefile containing:
all:;#echo $(filter jobserver,$(.FEATURES))
then run make with that makefile.
I'm trying to step through an OpenFOAM application (in this case, icoFoam, but this question is in general for any OpenFOAM app).
I'd like to use gdb to step through an analysis running in parallel (let's say, 2 procs).
To simply launch the app in parallel, I type:
mpirun -np 2 icoFoam -parallel
Now I want to step through it in gdb. But I'm having trouble launching icoFoam in parallel and debugging it, since I can't figure out how to set a break point before the application begins to execute.
One thing I know I could do is insert a section of code after the MPI_Initialize that waits (and endless loop) until I change some variable in gdb. Then I'd run the app in parallel, attach a gdb session to each of those PIDs, and happily debug. But I'd rather not have to alter the OpenFOAM source and recompile.
So, how can I start the application running in parallel, some how get it to stop (like at the beginning of main) and then step through it in gdb? All without changing the original source code?
Kindest regards,
Madeleine.
You could try this resource
It looks like the correct command is:
mpirunDebug -np 2 xxxFoam -parallel
There is the following process: NetAuthSysAgent which calls/create repeatedly the process mount_nfs of a non-existing NFS share. I have a hunch that these calls are triggered by another process which gives the instruction.
Therefore I would like to profile the NetAuthSysAgent, perhaps using dtrace, and see all inputs (like IPC incl. signals, queues, files, sockets, etc.) for this process and also verify all read files or memory area before it calls the mount_nfs process.
How can I do that? Is dtrace suitable and if yes how?
Any help is welcome. Even if it would only apply/work on FreeBSD or Solaris, I could try to adapt it myself to OS X.
Update:
I have been using the following commands yet, but I have covered only the file part:
dtrace -q -n syscall::open:entry'{ printf("%-16s %-16s\n",execname,copyinstr(arg0)); }'
the above command gave me the processes and opened files.
opensnoop -va -n NetAuthSysAgent
the above command is a script calling DTrace stuffs, it is more or less the same as the previous command but it is filtered by the process NetAuthSysAgent and provides a few more information (e.g. UID, PID, FD, etc.)