Resource-Efficient Deep Learning

As emerging deep neural network (DNN) models continue to grow in size, using large GPU clusters to train DNNs or run inference is becoming an essential requirement for achieving acceptable training/inference times. However, scaling DNN training to hundreds or thousands of GPU nodes poses a significant challenge to resource efficiency. Expensive GPUs are often wasted for many reasons: waiting for network communication, bottlenecked by memory bandwidth, or operators with not enough compute intensity to utilize all GPU cores. I am interested in building new systems for machine learning that leverage fast networking to improve the efficiency of large DNN model training and inference.

DeepPool: burst parallel training

At large scales, the conventional scaling strategy of increasing global batch size doesn’t reduce overall training time to accuracy. For continued improvement on time to accuracy, we must consider “strong scaling” strategies that hold the global batch size constant and allocate smaller batches to each GPU. Unfortunately, small-batch training often underutilizes modern GPUs with many compute cores. Thus, we have had to make an unfortunate choice between high cluster utilization or fast training.

DeepPool addresses this challenge by enabling both good cluster throughput and fast training. DeepPool incorporates two key ideas. First, a “burst parallel training planner” dynamically adjusts the number of GPUs allocated to each layer, so that layers with less parallelism can use fewer GPUs. This increases overall cluster efficiency because it frees up underutilized GPUs for use by other training tasks. Second, a new collection of GPU multiplexing techniques allows a background training task to reclaim the GPU cycles unused by the large-scale foreground task.

Cost-efficient MoE serving

More recently, we are exploring how asynchrony can make serving of Mixture-of-Experts models more cost-efficient. FluidEP (SOSP ’26) introduces fluidic expert parallelism: instead of lockstep all-to-all collectives, each device independently receives token dispatches via point-to-point transfers, begins expert computation as soon as a partial batch arrives, and returns results without waiting for global synchronization. On a 16×L40S cluster with 200 Gbps inter-node networking, FluidEP achieves up to 2.3× higher decoding throughput than state-of-the-art expert parallel serving — efficient MoE inference need not be confined to hyperscaler infrastructure.