Zero-Copy in Containers: Accelerating KubeVirt and Memory Providers

Addressing Virtualization Challenges

• 💡 Rising virtualization costs are pushing enterprises to migrate from traditional VM setups to cloud-native Kubernetes environments for infrastructure convergence and cost savings.
• 🚀 KubeVirt enables running virtual machines within Kubernetes Pods, allowing for unified management of both VM and container workloads.
• 🚧 The default KubeVirt data path, using V devices and network namespaces, introduces significant overhead, leading to suboptimal network performance.

Limitations of Existing Solutions

• 📉 Initial attempts with Netkit devices improved throughput by 30% and latency by 12% but were deemed insufficient for high-performance needs.
• 🚫 SR-IOV (SIOV) offers hardware virtualization for the fastest path but lacks host visibility for policy enforcement and Kubernetes service access, making it unsuitable for managed environments.
• ⚠️ Applications in containers with their own network namespaces cannot directly bind to physical NIC queues, posing a challenge for zero-copy mechanisms like AF_XDP and memory providers.

Introducing NIC Queue Leasing

• 🔑 The proposed solution involves leasing physical hardware queues from the NIC to virtual queues created within Netkit devices.
• 🔄 This allows containerized applications to bind to these virtual queues, which then transparently proxy traffic to the underlying physical queues.
• ✅ This mechanism enables zero-copy solutions such as AF_XDP for KubeVirt's QEMU backend and memory providers (io_uring, Devmem TCP for GPU memory) within containers.

Performance and Implementation Details

• 📊 Preliminary benchmarks using the QEMU AF_XDP backend with queue leasing showed a 37% improvement in throughput and 30-35% lower latency.
• 🛠️ The implementation introduces a new Q_CREATE API command with a lease attribute, allowing specification of the physical device and queue ID for leasing.
• 🔒 The system ensures proper lifetime management and prevents deadlocks by consistently locking virtual devices before physical ones, and blocking queue resizing on leased queues.

Extending the XDP API

• 🎯 To enhance KubeVirt, a new AF_XDP TX hook is needed for policy enforcement on egress traffic from VMs, complementing existing ingress policy capabilities.
• 📈 The XDP API is being revamped to support BPF multi-attach capabilities (similar to TCX), enabling multiple XDP programs and queue-range specific attachments.
• ⚙️ This revamp aims to provide a more flexible and powerful XDP framework for future networking needs, including dynamic program execution without extensive driver modifications.