GPU Optimization

Summary: GPUs are the primary cost center and performance lever for modern AI, where small inefficiencies compound across training runs and inference fleets to dramatically increase cloud bills, so optimization is a systems problem spanning code, memory, interconnect, and orchestration. Key dimensions include compute efficiency (SM occupancy and tensor‑core utilization), memory bandwidth and locality, kernel launch/scheduling overhead, inter‑GPU communication (NVLink/PCIe/NCCL), and orchestration/packing (instance sizing, MIG, multi‑tenant isolation); practical levers include mixed precision, operator fusion and memory-management strategies, kernel batching, optimized communication patterns, and improved instance/tenant packing.

Why GPU optimization matters now

GPUs are the primary cost center and performance lever for modern AI. Small inefficiencies compound across training runs and production inference fleets: 10% lower utilization can mean 30–50% higher cloud bill once wasted I/O, stalls, and retries are included. Optimization is not cosmetic it’s a systems problem spanning code, memory, interconnect, and orchestration.

Key dimensions of GPU performance

• Compute efficiency: SM (streaming multiprocessor) occupancy and tensor core utilization.
• Memory bandwidth and locality: DRAM, L2, cache, and host↔device transfers.
• Kernel launch and scheduling overhead: large numbers of small kernels create CPU-side bottlenecks.
• Inter-GPU communication: NVLink, PCIe, and the software stack (NCCL).
• Orchestration and packing: instance sizing, MIG, and multi-tenant isolation.

Practical optimization levers (technical)

Algorithmic and framework-level

• Mixed precision: use FP16/BF16 with dynamic loss-scaling; native AMP in PyTorch (torch.cuda.amp) or TensorFlow mixed_precision. Tensor cores unlock 4–8× higher throughput for compatible ops.
• Kernel fusion: combine pointwise ops (fused kernels, XLA, TorchScript custom ops) to reduce memory traffic and kernel launch overhead.
• cuDNN / cuBLAS tuning: allow autotuning (cudnn.benchmark=True in PyTorch) to pick best convolution algorithms, and control workspace size for large tensors.

Memory and data-movement

• Pinned host memory and async copies: use cudaHostAlloc / pinned buffers and cudaMemcpyAsync with multiple CUDA streams to overlap transfers with compute.
• Prefetching & Unified Memory: prefetch to GPU ahead of use; for very large models, considered unified memory with careful prefetch and async migration.
• Reduce fragmentation: use framework caching allocators or set PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb to avoid OOM overhead; avoid frequent cudaFree/cudaMalloc cycles.

Parallelism and scaling

• Data-parallel at scale: use NCCL-enabled DistributedDataParallel (PyTorch DDP), set NCCL_DEBUG=INFO for visibility, and bind NICs (NCCL_SOCKET_IFNAME) for RDMA.
• Model-parallel primitives: pipeline parallelism to hide communication, FSDP/ZeRO for memory savings, and activation checkpointing to trade recompute for memory.
• Overlap comm & comp: schedule gradient reductions on streams separate from compute, use asynchronous NCCL collectives when supported.

Orchestration and hardware

• Topology-aware placement: query nvidia-smi topo -m, place communicating ranks on NVLink-connected GPUs, avoid cross-socket PCIe when possible.
• MIG and instance sizing: For A100/MIG, partition GPUs to match workload latency/throughput; smaller inference pods can be more cost-efficient.
• Kubernetes: use NVIDIA device plugin and GPU operator; request GPUs explicitly and use node affinity to control topology.

Measurement and profiling (always measure before tuning)

• Tools: nsight-systems (nsys), nsight-compute (ncu), NVIDIA DCGM for telemetry, and framework profilers (PyTorch profiler with NVTX ranges).
• Key metrics: SM utilization, achieved occupancy, memory throughput (GB/s), warp execution efficiency, IPC, kernel launch latency, L2 hit rate, PCIe/NVLink throughput.
• Rule of thumb: SM utilization <70% or memory throughput << device peak indicates opportunities. Many “slow” workloads are memory-bound or IO-bound, not compute-bound.

Actionable checklist (startup-ready)

• Profile first: nsys profile --trace=cuda,nvtx -o trace python train.py; inspect SM and memory charts.
• Enable mixed precision: integrate torch.cuda.amp or TF mixed_precision; verify numeric stability with loss scaling.
• Batch sizing: scale batch until either memory or compute saturates; use gradient accumulation if GPU memory limits batch size needed for throughput.
• Overlap I/O: use DataLoader with pinned memory and multiple workers; move data transfers into separate streams.
• Tune NCCL and placement: set NCCL_DEBUG=INFO, NCCL_SOCKET_IFNAME to the right interface; colocate GPUs with NVLink when possible.
• Reduce kernel overhead: fuse element-wise ops, use JIT/custom kernels for hot paths.
• Use FSDP/ZeRO for very large models; combine with offload when memory is the bottleneck.
• Automate telemetry: ship DCGM metrics to your monitoring stack and set thresholds for SM util and memory throughput.

Final note

GPU optimization is multi-layered the highest returns come from measuring end-to-end, then iterating across kernel fusion, memory movement, communication, and orchestration. Treat each training job as a mini-systems project: profile, hypothesize, change one variable, and validate cost/perf tradeoffs in production-like conditions.