Pipely Launch: Kaizen 20 Live in Denver with Gerhard Lazu

The Genesis of Kaizen and Pipely

• 💡 Kaizen, a concept of consistent improvement, has been a part of The Changelog for about four years, evolving from "Ship It" episodes to a public, in-person event.
• 🚀 Pipely, initially a "pipe dream" born from a frustration with Fastly's Varnish configuration, began its development 18 months ago.
• 🤝 The journey of Pipely was fueled by collaboration and friendship, with the goal of improving something in public to inspire others.

Addressing Performance Bottlenecks

• 🎯 A significant issue identified was a low cash hit ratio (17.93%) on the current production CDN, leading to slow load times for users.
• 💰 Past issues with S3 costs were resolved by migrating to R2, drastically reducing expenses.
• ⚠️ A critical incident involved an instance crashing due to out of memory errors, highlighting the need for better memory allocation management for Varnish.

Pipely's Live Launch and Technical Deep Dive

• 🚀 The culmination of the Pipely project was its live launch on stage during Kaizen 20, marking a transition from Fastly to the new Pipely infrastructure.
• 💻 Gerhard demonstrated live coding to increase Varnish memory allocation, resolving the crash and showcasing the team's ability to iterate quickly.
• ⚡ The deployment pipeline was highlighted as a key improvement, reducing deployment time from 20 minutes to approximately two and a half minutes.

Infrastructure and Technology

• 🌐 Pipely is built on open-source technologies, specifically Varnish Cache, emphasizing the team's commitment to open source.
• 🔗 The project utilizes Dagger for its remote engine, enabling reproducible testing and development environments that closely mirror production.
• 🔒 A custom TLS Exterminator solution, written in Go, was developed to handle SSL termination for backends, a capability not natively supported by open-source Varnish.

Performance Improvements and Future

• ✅ Post-launch, the cash hit ratio for the homepage improved significantly to 99.1%, demonstrating the effectiveness of Pipely.
• 🌍 DNS records were updated to route all traffic to Pipely, marking a successful transition and farewell to Fastly.
• 📊 Monitoring through Honeycomb allows for detailed analysis of requests, data centers, and potential issues like bot traffic or hotlinking.
• 📈 Future improvements include exploring disk caching for Varnish and optimizing instance sizes based on regional traffic patterns.