MPI Buffers for Halo Exchanges

Halo exchanges typically involve slices of arrays. In our current example, we are discussing exchanging parts of a two-dimensional array. We have learned how to communicate one-dimensional arrays; now we must consider communicating portions of higher-dimensional arrays.

As we learned, MPI does not construct buffers from variables; it reads a specified number of bytes starting at a specified location in memory. We must therefore take into consideration how arrays are organized in memory.

In a given programming language, an array can be row major oriented or column major oriented. In a row-major oriented language (most of them, including C/C++ and Python) a two-dimensional array is represented in memory in terms of its indices as

That is, the second index changes most rapidly as we traverse the array in memory.

In C/C++ the rows may or may not be contiguous in memory, but the elements of each row will be contiguous.

In a column-major oriented language (Fortran and some others such as Matlab, Julia, and R) the layout is by column. These languages also count from 1, at least by default, as represented below.

In this case the first index varies most rapidly. In Fortran all columns of a two-dimensional array (and analogously for higher-dimensional arrays) are guaranteed to be contiguous in memory.

The figure we examined previously illustrates an exchange of columns and would be used for the column-major languages. To visualize the exchange for row-major languages, we just rotate that figure 90 degrees.