MPI and Threads

In our introductory sections, we discussed distributed and shared memory. We have focused so far on distributed-memory programming. But shared-memory programming is widely used, particularly with the increase in utilization of local shared-memory resources such as graphical processing units (GPUs). It is possible to use both in a single program; this is often called hybrid parallel programming.

It is possible to use MPI in a shared-memory model but this is a more advanced topic than we will cover here. The most common applications are MPI with a shared-memory library such as OpenMP. Each process invokes the threading library while MPI handles interprocess communications.

Shared memory introduces some new complications that we will consider in the next sections. Suffice it to say for now that any correct program that uses shared memory techniques must be thread safe. In order to make MPI compatible with this, some new features have been introduced into the standard, particularly in how MPI is initialized.

We have only used MPI_Init so far. But now we have MPI_Init_thread. Threads can be initialized as single, meaning there will be only one thread in the program. This is equivalent to MPI_Init and the standard now requires that an implementation treat MPI_Init as MPI_Init_thread in “single thread” mode.

Another mode is funneled. In this case, only the master thread makes any MPI calls, and it does not invoke MPI within any SMP parallel region.

We can also allow for serialized threading. This does not mean that the processes do not use threads – that is single – but that only one thread can make MPI calls at a time.

Finally, MPI allows for multiple threads to make MPI calls at any time, subject to some conditions. A standards-compliant, thread-safe MPI implementation must provide a thread_multiple capability. However, implementations are not required to be thread-safe; the minimum support is thread_single (i.e no threading). But the major implementations (MPICH and its derivatives, OpenMPI) are thread safe.

C++

MPI_Init_thread(&argc, &argv, required, &provided);

Fortran

! mpi_err is optional as usual, if including mpi_f08 module
INTEGER required, provided, mpi_err
call MPI_INIT_THREAD(required, provided, mpi_err)

In C/C++ and Fortran, required and provided are integers, but required has values provided by the MPI header:

MPI_THREAD_SINGLE
MPI_THREAD_FUNNELED
MPI_THREAD_SERIALIZED
MPI_THREAD_MULTIPLE

The return value provided is the level of thread support actually provided by the implementation for this procedure.

Any C/C++ or Fortran code that calls MPI_Init should assume that no threading is used. If a code may become hybrid, MPI_Init_thread can be used with MPI_THREAD_SINGLE. This will be equivalent to MPI_Init but will “future proof” it. The required value can easily be changed.

Python

The mpi4py implementation we have used in our examples automatically initializes MPI when the module is imported. If we attempt to call MPI.Init_thread explicitly, we may encounter an error, since MPI_Init cannot be invoked in any form more than once per program. Mpi4py always by default initializes MPI with MPI_Init_thread specifying MPI.THREAD_MULTIPLE by default. Changes must be made throut the mpi4py.rc object.

import mpi4py
mpi4py.rc.thread_level="single"

Examples for each language

#include <iostream>
#include <mpi.h>

using namespace std;

int main(int argc, char* argv[]) {
    int provided;
    MPI_Init_thread(&argc, &argv, MPI_THREAD_MULTIPLE, &provided);
    cout<<"Requested "<<MPI_THREAD_MULTIPLE<<" Provided "<<provided<<endl;
    MPI_Finalize();

    exit(0);
}

program mpi_threads
use mpi_f08

integer ::  provided

   call MPI_INIT_THREAD(MPI_THREAD_MULTIPLE, provided)
   write(*,'(a,i2,a,i2)') "Requested ",MPI_THREAD_MULTIPLE," Provided ",provided
   call MPI_Finalize()

end program

import mpi4py
provided=mpi4py.rc.thread_level
#Reference:https://mpi4py.readthedocs.io/en/stable/mpi4py.html#mpi4py.mpi4py.rc

from mpi4py import MPI

thread_levels=["single", "funneled", "serialized", "multiple"]
mpi_levels=[MPI.THREAD_SINGLE,MPI.THREAD_FUNNELED,MPI.THREAD_SERIALIZED,MPI.THREAD_MULTIPLE] 
for t in range(len(thread_levels)):
    print(t,mpi_levels[t],thread_levels[t])

print(f"Requested:{MPI.THREAD_MULTIPLE} Provided:{provided}")