GPU Programming Model

The accelerator is a type of coprocessor, a unit separate from the CPU that carries out certain computations. Hence it is necessary to transfer data from main CPU memory to the accelerator memory.

We will use CUDA terminology in this discussion since it is the most widely used lower-level programming library for GPUs.

The CPU is called the host and the accelerator is the device. Code to be executed on the device is the device code. A function executed on the device is called a kernel. The host code launches a kernel by transferring the data to the device. The kernel is responsible for launching threads.

From the programming perspective, thread blocks are organized into grids which can be 1, 2, or 3 dimensional. Kernels are launched with an execution configuration specifying the grid layout. It is not always necessary for most programmers to be concerned about how grids map to SMs, but SM assignments and other hardware details can optionally be included in the execution configuration.

As already mentioned, all threads in a thread block run on the same SM, but the independence of the SMs means that blocks must not have dependencies on other blocks.

The transfer of data between the host and device is the major bottleneck in GPU programming and should be minimized.

Programming GPGPUs

GPUs are SIMD (single-instruction multiple data) systems. They are programmed with a threading model. The limited instruction set and SIMD model means that GPUs are typically programmed by launching multiple kernels within a program. (Some newer NVIDIA models have the ability to handle multiple streams of instruction, but this is only for some architectures and only available in CUDA.)

Programming systems can be broken into three basic categories. Below are some examples of each type.

Low-level libraries/languages

1. CUDA. NVIDIA only.
2. HIP/HPCC/ROCM AMD product intended to be generic but aimed at their hardware.
3. SYCL C++ library intended to abstract the device. Successor to Apple’s OpenCL.
4. Metal Apple’s library for their hardware. Aimed at machine learning but can also handle graphics and gaming.

Intermediate libraries/directives

1. OpenMP. OpenMP has been extended considerably to enable it to work with devices.
2. OpenACC. NVIDIA directives-based library.

High-level language support

1. CUDA Python and Numba.
2. JuliaGPU for the Julia language.
3. torch.compile Interface for kernel optimization for the Torch machine-learning system.