Inside Canada's Fastest Supercomputer: Direct Die Liquid Cooling & 82 Million Dollar Build

Supercomputer Overview and Scale

• 🚀 The video explores Canada's fastest supercomputer at Simon Fraser University, an $82 million CAD (approx. $60 million USD) facility.
• 💡 This powerhouse is designed to support research in fields ranging from AI to zoology for tens of thousands of scientists.
• 🧠 The supercomputer features a massive scale with 165,000 CPU cores and 20 million GPU cores, complemented by a petabyte of RAM.

GPU Compute Nodes and Cooling

• ⚡ A significant portion of the supercomputer comprises 640 Nvidia H100 GB GPUs, each costing around $31,000 USD.
• 💧 The nodes utilize a direct die liquid cooling system, achieving over 90% heat capture, a substantial improvement over traditional methods.
• 🧊 Every component, including CPUs, GPUs, VRMs, network interfaces, and even system memory, is directly liquid-cooled, eliminating the need for fans.
• 🌡️ The cooling loop involves complex manifold systems for GPUs, networking boards, power delivery, RAM, and CPUs, with specific routing for optimal heat dissipation.

CPU Nodes and Networking

• ⚙️ The supercomputer also includes 192-core CPU nodes (Zen 5 EPYC Turing cores) with 768 GB of memory per node, totaling nearly 400 cores per 1U unit.
• 🌐 Networking is handled by 200 gigabit connections with NDR, dynamically shared between nodes, and inter-rack connections of 800 gigabit for island clusters.
• 💾 Specialized nodes include 49 PB of spinning rust storage and 2 PB of NVMe storage, with some nodes offering up to 8 TB of RAM for specific jobs.

Data Center Infrastructure and Cooling Systems

• 🏭 The data center infrastructure includes racks fed by dual three-phase 60 amp feeds, capable of up to 70,000 watts per rack.
• 💧 A sophisticated cooling distribution system (CDU) from Vertiv manages up to 600,000 watts of cooling capacity per unit, using a liquid-to-liquid heat exchanger.
• 💨 Evaporative cooling towers provide the primary chilled water supply, augmented by mechanical chillers for times when ambient temperatures exceed 33°C.
• 🔊 Acoustic damping is applied to the cooling towers, and the facility's two-foot-thick concrete floor deflects half an inch under the weight of the equipment.

Funding and Partnerships

• 💰 The $82 million CAD budget was funded through contributions from the Digital Research Alliance of Canada, BCKDF, and vendor contributions (significant discounts).
• 🤝 Key partners and contributors mentioned include MSI, Lenovo, DDN, and Vertiv for cooling solutions.
• 💧 The facility uses AquaM tubing for its coolant lines, chosen for its durability and resistance to rust compared to stainless steel.