University of Michigan - Ann Arbor

In an edge system where direct CPU expert evaluation and direct memory access (DMA) transfers across PCIe draw from the same host DRAM subsystem, a saturated PCIe transfer leaves limited memory bandwidth for CPU processing. If $$B_H$$ denotes the measured host-side expert-processing bandwidth and $$B_P$$ denotes the pinned expert-transfer bandwidth across PCIe, the residual host bandwidth $$B_R$$ available for concurrent in-place CPU expert computation is: $$B_R = \max(B_H - B_P, 0)$$ When $$B_H > B_P$$, the CPU can execute missed experts using this residual bandwidth without throttling the saturated PCIe transfer.

Residual Host Bandwidth Formulation for CPU Expert Execution

Order the steps to determine and utilize residual host bandwidth for CPU expert execution under a saturated PCIe bus.

For $$m$$ missing experts of size $$S$$, let $$q$$ experts use the concurrent PCIe cache-fill branch and $$m-q$$ execute on the CPU. When $$B_H>B_P$$, their approximate execution times are $$T_{\text{fill}}(q)\approx\frac{qS}{B_P}$$ and $$T_{\text{cpu}}(m-q)\approx\frac{(m-q)S}{B_H-B_P}$$. Balancing the branches gives $$\frac{q}{m-q}\approx\frac{B_P}{B_H-B_P}$$ and therefore $$q^*\approx m\frac{B_P}{B_H}$$. The runtime rounds $$q^*$$ to an integer and retains at least one cache fill so that the cache continues warming. As $$B_H$$ approaches $$B_P$$ from above, $$q^*$$ approaches $$m$$, reducing the policy to pure on-demand cache fill.

Optimal MoE Decode Cache-Fill Count ($$q^*$$ Policy)

Calculate the residual host bandwidth (B_R) available for concurrent in-place CPU computation, and explain whether the CPU can execute missed experts concurrently without contending with or throttling the PCIe bus.

Learn Before

Related