University of Michigan - Ann Arbor

At small batch sizes typical of edge decoding, Mixture-of-Experts (MoE) expert computation is memory-bandwidth-bound because each token requires streaming the full weight tensors of its routed experts once. Consumer CPUs connected through dual-channel DRAM have peak memory bandwidths of approximately $$50\text{ GB/s}$$ with DDR4 and $$80\text{--}90\text{ GB/s}$$ with DDR5, whereas modern discrete GPUs achieve $$1.0\text{--}1.8\text{ TB/s}$$ from on-package VRAM. This host-memory bandwidth constraint limits the throughput of missed experts executed entirely on the CPU to a small fraction of the GPU's potential, regardless of the number of available CPU cores.

Host DRAM Bandwidth Bottleneck in CPU Expert Execution

Match each concept related to edge MoE execution with its accurate description.

Explain why allocating additional CPU cores fails to resolve the decode throughput bottleneck during CPU expert execution.

When an active expert is not present in GPU VRAM during decoding, the system can either transfer its weights across PCIe to execute on the GPU, or compute it directly on the CPU where the weights reside in host RAM. Neither extreme is universally optimal: exclusively moving weights over PCIe leaves available host DRAM bandwidth and CPU cores idle if host memory can supply more data than the bus can move; conversely, computing entirely on the CPU leaves the PCIe interconnect idle and forfeits future cache hits that a VRAM cache fill provides. Because the balance between host memory bandwidth and PCIe transfer bandwidth varies widely across machines (such as LPDDR5 laptops versus high-speed PCIe desktops), the optimal division of miss handling cannot be statically predetermined and must be tuned to specific hardware resources.

Learn Before

Related