GPU-Disaggregated Serving for Deep Learning Recommendation Models at Scale

Challenges in DLRM Serving

• ⚠️ Deep Learning Recommendation Models (DLRMs) require high CPU and memory but low GPU, leading to inefficient resource provisioning in production.
• 💡 Resource fragmentation is a severe issue in shared clusters, causing scheduling failures despite sufficient quotas due to the high CPU-to-GPU ratio of DLRM inference instances.
• 📈 Seasonal load spikes, such as e-commerce promotion events, lead to significant underutilization if provisioned for peak, and effective capacity loaning from training clusters is hindered by CPU-to-GPU ratio mismatches.

Prism: A GPU-Disaggregated Solution

• 🚀 Prism is a production DLRM serving system, a collaborative project between HKUST and Alibaba Group, designed to eliminate GPU fragmentation through resource disaggregation.
• 🧩 It automatically partitions DLRMs into CPU-intensive and GPU-intensive subgraphs, scheduling them on dedicated CPU nodes (CNs) and heterogeneous GPU nodes (HNs) respectively.
• ✅ This approach establishes two independently scalable resource pools interconnected by a high-speed RDMA network, enabling efficient resource sharing and utilization.

Architectural Design and Components

• 🧠 Prism ensures transparency for model developers by integrating graph partitioning and disaggregation optimization as a final stage in the existing workflow, without requiring model modifications.
• 🛠️ Key components include a heuristic graph partitioning approach based on offline profiling to split GPU subgraphs and characterize operators.
• ⚡ A unified operator (fusgraph op) handles RPCs, aggregates tensors, and leverages GPU direct RDMA for zero-copy data transmission between CNs and HNs.
• 🎯 The resource manager employs topology-aware scheduling, prioritizing co-location of CN and HN instances within the same SU and limiting node-level deployment density to prevent network saturation.
• 📊 An SLO-aware communication scheduler extends the native RDMA stack, implementing incast control by adapting window size and reordering requests to maintain service performance.

Impact and Benefits

• 📉 Prism significantly reduces CPU fragmentation by 53% and GPU fragmentation by 27% in crowded GPU clusters, utilizing fragmented resources more effectively.
• 💰 It enables capacity loaning from training clusters during peak seasonal traffic, saving over 90% of GPUs by allowing recommendation models to efficiently use training nodes.
• ✅ Deployed in Alibaba's production clusters for over two years, running on 10,000+ GPUs, demonstrating its scalability and reliability in real-world deployments.
• ⚡ Even for models with the largest communication volumes, Prism maintains service performance with at most 6% performance loss.

Future Perspectives

• 💡 Disaggregation is a philosophical approach applicable to any workload with heterogeneous resource requirements during inference, transforming resource provisioning.
• ⚠️ Disaggregation introduces the network as a new resource dimension, necessitating better performance isolation at both node and network switch levels.
• 📈 Higher deployment density on training nodes requires more fine-grained management to reduce the impact of single point failures on service availability.