Higher Dimensional Data Decomposition

We chose to divide our computational grid into rows for C++ and Python and columns for Fortran because of the memory layouts used by the three languages. But it could be that a one-dimensional decomposition would not be sufficient to allow us to complete the solution in a reasonable amount of time or memory utilization, and we will need a two-dimensional decomposition.

Since columns in C++ and Python and rows in Fortran are not contiguous in memory, in order to create an MPI buffer we must skip through the elements and copy the appropriate ones into the buffer.

C++

  int nrows=nrl+2
  int ncols=ncl+2
  double *buf=new double[nrows];
  for (i = 0; i < nrows; i++ ) {
      [buf[i]=u[i%ncols][0];
  }

Python

nrows=nrl+2
ncols=ncl+2
buf=np.zeros(nrows)
for i in range(nrows):
    buf[i]=u[i%ncols,0]

Fortran

!declare buff allocatable(:)
    nrows=nrl+2
    ncols=ncl+2
    allocate(buf(0:ncols-1)
    do i=0,ncols-1
        buff(i)=u(0,mod(i,nrows))
    enddo