Building a Custom CDN: Pipely's Journey and Challenges

The Pipely Project: A CDN Under Construction

• 🚀 Gerhard Lazu details the extensive work on Pipely, a custom Content Delivery Network (CDN) being built for changelog.com.
• 💡 The project aims to improve performance and control over content delivery, moving away from reliance on third-party CDNs.
• 🎯 The goal is to have a production-ready CDN by the next Kaizen episode, with a focus on stability and efficiency.

Performance Benchmarking and Challenges

• 📊 Benchmarking reveals that the current CDN (Fastly) is faster than the new Pipely CDN, but the bottleneck is often the local network connection.
• ⚠️ A significant challenge identified is the low cache hit rate (around 18%) on the current CDN for application requests, defeating its purpose.
• 📈 Pipely's CDN aims for a higher cache hit rate, serving content from the edge consistently, even if stale.

Debugging and Rollout Issues

• 🐛 Several bugs were encountered during the testing phase, including incorrect URL routing, issues with background job processing, and problems with Slack notifications.
• 🛠️ These issues highlight the complexities of managing multiple application instances and ensuring consistent data and configuration.
• 🔄 The team is working on a strategy to mitigate these issues, including disabling background jobs on experimental instances and ensuring proper URL handling.

Technical Deep Dive: Varnish and Infrastructure

• ⚙️ The Pipely CDN utilizes Varnish Cache (open-source) and custom-built components to replicate features found in enterprise solutions.
• 🌐 The infrastructure involves multiple processes for TLS termination, static assets, and the application origin, all managed locally for testing.
• 🧪 Rigorous testing, including Varnish Test Cases (VTC) and acceptance tests with Hurl, is crucial for validating the CDN's behavior and correctness.

Future Development and Launch Party

• 📈 Remaining tasks include forwarding logs to Honeycomb and S3, implementing content purging, and refining edge redirects.
• 🤝 The team emphasizes the importance of patience, diligence, and collaboration in building such a complex system.
• 🎉 A potential launch party in Denver is proposed for Kaizen 20 to celebrate the completion of the Pipely project.