As emerging deep neural network (DNN) models continue to grow in size, using large GPU clusters to
train DNNs or run inference is becoming an essential requirement for achieving acceptable
training/inference times. However, scaling DNN training to hundreds or thousands of GPU nodes poses
a significant challenge to resource efficiency. Expensive GPUs are often wasted for many reasons:
waiting for network communication, bottlenecked by memory bandwidth, or operators with not enough
compute intensity to utilize all GPU cores. I am interested in building new systems for machine
learning that leverage fast networking to improve the efficiency of large DNN model training and
inference.
DeepPool: burst parallel training
At large scales, the conventional scaling strategy of increasing global batch size doesn’t reduce
overall training time to accuracy. For continued improvement on time to accuracy, we must consider
“strong scaling” strategies that hold the global batch size constant and allocate smaller batches
to each GPU. Unfortunately, small-batch training often underutilizes modern GPUs with many compute
cores. Thus, we have had to make an unfortunate choice between high cluster utilization or fast
training.
DeepPool addresses this challenge by enabling both good cluster throughput and fast training.
DeepPool incorporates two key ideas. First, a “burst parallel training planner” dynamically adjusts
the number of GPUs allocated to each layer, so that layers with less parallelism can use fewer
GPUs. This increases overall cluster efficiency because it frees up underutilized GPUs for use by
other training tasks. Second, a new collection of GPU multiplexing techniques allows a background
training task to reclaim the GPU cycles unused by the large-scale foreground task.
StreamEP: MoE decoding without communication barriers
Mixture-of-Experts (MoE) models are served with expert parallelism (EP), which spreads experts
across GPUs and shuffles tokens among them with collective communication at every layer. Each
collective is a barrier: every GPU waits for the slowest transfer or the most loaded device before
it can move on. Hyperscalers hide these stragglers with over-provisioned interconnects and
MoE-optimized software stacks that require Hopper-generation GPUs. On commodity GPU clusters with
shared and nonuniform interconnects, the barrier serializes execution behind the slowest transfer.
StreamEP (SOSP ’26) removes the barrier entirely. Each GPU independently receives tokens,
executes the appropriate layer, and forwards results as soon as they are ready. Our serving system,
StreamInfer, realizes StreamEP on vanilla NCCL with per-layer token queues, a defragmenting
scheduler that consolidates partial batches while guaranteeing forward progress, and a two-phase
communicator for asynchronous, variable-size GPU-to-GPU transfers. On 16×L40S and 16×A100 clusters
with 200 Gbps inter-node networking, StreamInfer achieves up to 1.7× higher decoding throughput
than traditional EP, showing that efficient large-scale MoE serving is achievable on commodity
hardware.
Moebius: switching parallelism at runtime
The best way to split an MoE model across GPUs depends on how many requests are in flight: tensor
parallelism (TP) is faster at low concurrency, and expert parallelism (EP) wins at high
concurrency. Real workloads cross this boundary all the time. Online serving sees bursts of
arrivals followed by quiet periods, and reinforcement-learning rollouts start as a high-concurrency
burst that decays into a long tail of stragglers. A fixed layout loses performance whenever the
workload moves to the other side.
Moebius (preprint) switches between EP and TP at runtime without restarting the engine or
dropping in-flight requests. EP and TP compute the same function over byte-identical expert weights
and KV cache, so a switch changes only which GPU owns each slice. Moebius moves just those slices
between decode steps, using fused GPU-to-GPU transfer kernels that reshard a single copy of the
weights and KV cache at fixed memory addresses, so captured CUDA graphs stay valid. On 8×H200 GPUs
serving Qwen3-235B-A22B, Moebius matches the better static layout at every operating point and
beats it by 1.16–1.25× on RL rollouts. Each switch takes 215–434 ms, and keeping both layouts
resident costs only 2.4% extra memory.
