Using code with underlying parallelism in OpenMDAO - parallel-processing

Question

I am interested in adding an underlying parallelism to a couple of our OpenMDAO components. The bulk of the code in these components is written in Fortran. The Fortran code is wrapped in python and then used as python modules in OpenMDAO. I would like to run these Fortran codes in parallel using either OpenMP or OpenMPI. We are already planning to use the built-in parallel functionality of OpenMDAO, so this would be a second layer of parallelism. Would this be feasible? If so, do you have a recommended approach that would work well with OpenMDAO?

Answer 1

First I'll address the question about OpenMP. Currently OpenMDAO doesn't use OpenMP itself, and we don't have any plans to change that any time soon. So that means, we the framework doesn't really know or care if you happen to use it in your Fortran code at all. Feel free, with all the normal caveats about MPI + OpenMP codes in effect of course!
If you would like to use MPI parallelism in a component itself, that is directly supported by OpenMDAO. We have a fairly simple tutorial up for this situation, where the component itself would like to ask for multiple processors. Notable features of this tutorial are where the component asks the framework for more than one processor:
def get_req_procs(self):
"""
min/max number of cpus that this component can use
"""
return (1,self.size)
In this case, the component will accept anywhere from 1 proc, up to the number of elements in its array. In your case, you might want to restrict that to a single value, in which case you can return a single integer.
The other notable part is:
def setup_distrib_idxs(self):
"""
specify the local sizes of the variables and which specific indices this specific
distributed component will handle. Indices do NOT need to be sequential or
contiguous!
"""
comm = self.comm
rank = comm.rank
#NOTE: evenly_distrib_idxs is a helper function to split the array up as evenly as possible
sizes, offsets = evenly_distrib_idxs(comm.size, self.size)
local_size, local_offset = sizes[rank], offsets[rank]
self.local_size = int(local_size)
start = local_offset
end = local_offset + local_size
self.set_var_indices('x', val=np.zeros(local_size, float),
src_indices=np.arange(start, end, dtype=int))
self.set_var_indices('y', val=np.zeros(local_size, float),
src_indices=np.arange(start, end, dtype=int))
This code tells the framework how your distributed data gets split up across the many procs. The details of this methods are going to vary heavily from one implementation to the next. In some cases, you might have all procs have all the data. In others (like this one) you'll distribute the data evenly across the procs. In still other cases you might have a combination of global and distributed data.
If you planned to use only OpenMP, you would probably share all the data amongst all the processes, but still request more than 1 proc. That way you ensure that OpenMDAO allocated enough procs to your comp that it can be useful in a multi-threaded context. You'll be handed a comm object that you can work with to divy up the tasks.
If you planned to use purely MPI, its likely (though not certain) that you'll be working with distributed data. You'll still want to request more than 1 proc, but you'll also have to split up the data.
If you decide to use OpenMP and MPI, then likely some combination of distributed and shared data will be needed.