Setting Up and Running MPI

We are now ready to write our first MPI program. Select your choice of language below for this example.

A Hello World example.

#include <iostream>
#include "mpi.h"

using namespace std;

int main(int argc, char *argv[]) {
    int rank, npes;

    MPI_Init(&argc, &argv);
    MPI_Comm_size(MPI_COMM_WORLD, &npes);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    if ( rank == 0 ) {
        cout<<"Running on "<<npes<<"  Processes\n";
    }
    cout<<"Greetings from rank "<<rank<<"\n";

    MPI_Finalize();

}

program hello
use mpi

    integer :: myrank, nprocs
    integer :: err

    call MPI_INIT(err)
    call MPI_COMM_RANK(MPI_COMM_WORLD, myrank, err)
    call MPI_COMM_SIZE(MPI_COMM_WORLD, nprocs, err)

    if ( myrank == 0 ) then
        print *, 'Running on ',nprocs,' Processes'
    endif

    print *, 'Greetings from process ', myrank
    call MPI_FINALIZE(err)


end program

from mpi4py import MPI
import sys

myrank = MPI.COMM_WORLD.Get_rank()
nprocs = MPI.COMM_WORLD.Get_size()

if myrank == 0:
    sys.stdout.write("Running on %d processes\n" %(nprocs))

sys.stdout.write("Greetings from process %d\n"%(myrank))

Preparing and Running MPI Codes

On a Remote Cluster

Refer to the instructions from your site, for example UVA Research Computing for our local environment. Nearly always, you will be required to prepare your code and run it through a resource manager such as Slurm.

For Python, you will need to install mpi4py. You may wish to create a conda environment for it. On the UVA system you must use pip rather than conda.

module load gcc openmpi
module load anaconda
pip install --user mpi4py

On a Local Computer

If you have access to a multicore computer, you can run MPI programs on it.

If using a compiled language, before you can build MPI codes you must install a compiler, and possibly some kind of IDE. See our guides for C++ or Fortran.

For Python, on all operating systems install mpi4py. To install mpi4py you must have a working mpicc compiler. For Anaconda this generally requires installing the gcc_linux-64 package using conda. Instructions are here for Linux and command-line MacOS. For Windows use the Anaconda package manager interface and install m2w64-gcc.

The author of mpi4py recommends using pip even with an Anaconda environment. This command will be similar on a local system to that used for installation on a multiuser system.

python -m pip install mpi4py

This may avoid some issues that occasionally arise in prebuilt mpi4py packages. Be sure that an appropriate mpicc executable is in the path.

Linux

Parallel programs are most frequently used in a high-performance computing environment such as the clusters we have discussed, but many multicore workstations are available that can run Linux and many parallel programmers are familiar with this environment. The simplest way to install MPI is to use a precompiled version for your distribution and compiler.

GCC The recommended MPI distribution is OpenMPI. Most distributions provide a package.

Intel oneAPI Installing the HPC Toolkit will also install IntelMPI.

NVIDIA HPC SDK The NVIDIA software ships with a precompiled version of OpenMPI.

The headers and libraries for MPI must match. Using a header from one MPI and libraries from another, or using headers from a version from one compiler and libraries from a different compiler, usually results in some difficult-to-interpret bugs. Moreover, the process manager must be compatible with the MPI used to compile the code. Because of this, if more than one compiler and especially more than one MPI version is installed, the use of modules ( environment modules or lmod becomes particularly beneficial. Both Intel and NVIDIA provide scripts for the environment modules package (lmod can also read these), with possibly some setup required. If you plan to use mpi4py as well as compiled-language versions, creating a module for Anaconda would also be advisable.

Mac OS

GCC Installing homebrew is the simplest way to set up MPI on a Mac. Install the gcc package followed by the open-mpi package.

Intel oneAPI Install the HPC Toolkit for IntelMPI.

NVIDIA HPC SDK The NVIDIA suite is not available for Mac OS.

Windows

GCC The simplest way to use OpenMPI on Windows is through Cygwin. In this case, the gcc compiler suite would first be installed, with g++ and/or gfortran added. Then the openmpi package could also be installed through the cygwin package manager.

Intel oneAPI Install the HPC Toolkit.

NVIDIA HPC_SDK Download the package when it is available.

MPI codes must generally be compiled and run through a command line on Windows. Cygwin users can find a variety of tutorials online, for example here.

The Intel oneAPI Basic Toolkit includes a customized command prompt in its folder in the Apps menu.

Build It

In a terminal window on the frontend rivanna.hpc.virginia.edu run

module load gcc openmpi

For Python add

module load anaconda

This will also load the correct MPI libraries. You must have already installed mpi4py. Activate the conda environment if appropriate.

Use mpiexec and –np only on the frontends! Use for short tests only!

Compiling C

mpicc –o mpihello mpi1.c

Compiling C++

mpicxx –o mpihello mpi1.cxx

Compiling Fortran

mpif90 –o mpihello mpi1.f90

Execute it

C/C++/Fortran

mpiexec –np 4 ./mpihello

Python

mpiexec –np 4 python mpi1.py

Submit It

Write a SLURM script to run your program. Request 1 node and 10 cores on the standard partition. The process manager will know how many cores were requested from SLURM.

srun ./mpihello

Or

srun python mpihello\.py

Using the Intel Compilers and MPI

Intel compilers, MPI, and math libraries (the MKL) are widely used for high-performance applications such as MPI codes, especially on Intel-architecture systems. The appropriate MPI wrapper compilers are

#C
mpiicc -o mpihello mpi1.c
# C++
mpiicpc -o mpihello mpi1.cxx
# Fortran
mpiifort -o mpihello mpi1.f90

Do not use mpicc, mpicxx, or mpif90 with Intel compilers. Those are provided by Intel but use the gcc suite and can result in conflicts.