Numba CUDA Example

A Numba CUDA example for fast matrix multiplication is part of the documentation distributed with the package.

from numba import cuda, float32
import numpy as np
import math

# Controls threads per block and shared memory usage.
# The computation will be done on blocks of TPBxTPB elements.
TPB = 16

@cuda.jit
def fast_matmul(A, B, C):
    # Define an array in the shared memory
    # The size and type of the arrays must be known at compile time
    sA = cuda.shared.array(shape=(TPB, TPB), dtype=float32)
    sB = cuda.shared.array(shape=(TPB, TPB), dtype=float32)

    x, y = cuda.grid(2)

    tx = cuda.threadIdx.x
    ty = cuda.threadIdx.y
    bpg = cuda.gridDim.x    # blocks per grid

    if x >= C.shape[0] and y >= C.shape[1]:
        # Quit if (x, y) is outside of valid C boundary
        return

    # Each thread computes one element in the result matrix.
    # The dot product is chunked into dot products of TPB-long vectors.
    tmp = 0.
    for i in range(bpg):
        # Preload data into shared memory
        sA[tx, ty] = A[x, ty + i * TPB]
        sB[tx, ty] = B[tx + i * TPB, y]

        # Wait until all threads finish preloading
        cuda.syncthreads()

        # Computes partial product on the shared memory
        for j in range(TPB):
            tmp += sA[tx, j] * sB[j, ty]

        # Wait until all threads finish computing
        cuda.syncthreads()

    C[x, y] = tmp

x_h = np.arange(16).reshape([4, 4])
y_h = np.ones([4, 4])
z_h = np.zeros([4, 4])

x_d = cuda.to_device(x_h)
y_d = cuda.to_device(y_h)
z_d = cuda.to_device(z_h)

threadsperblock = (TPB, TPB)
blockspergrid_x = math.ceil(z_h.shape[0] / threadsperblock[0])
blockspergrid_y = math.ceil(z_h.shape[1] / threadsperblock[1])
blockspergrid = (blockspergrid_x, blockspergrid_y)

fast_matmul[blockspergrid, threadsperblock](x_d, y_d, z_d)
z_h = z_d.copy_to_host()
print(z_h)
print(x_h @ y_h)

Discussion

This example sets the threads per block to 16 and we divide the computation into TPBxTPB chunks.

We import Numba CUDA with the cuda.jit decorator. We then establish the grid and assign thread blocks to it. Each thread block will compute an element.

The shared array is allocated on the device in a memory level that can be accessed by all threads in a block. This is faster than transferring data or allocating on the global device memory.

In the main program, we create three arrays on the host and then use to_device to copy them to the device global memory. The device arrays are passed to the fast_matmul function, which performs the computation. We then copy only the result back to the host with copy_to_host.

The @ operator was introduced in NumPy 1.10.0 for two-d matrix multiplication. For those not familiar with it, it is “syntactic sugar” for numpy.matmul.

Full documentation for Numba CUDA is available from NVIDIA.