Resiliency at Shopify with Camilo Lopez and Tai Dickerson

Carter Morgan and Stephanie Wong host Shopify guests Camilo Lopez and Tai Dickerson this week. Shopify streamlines the online purchasing process so merchants and customers can transact with confidence.

Camilo and Tai talk in-depth about Shopify's tech stack and why the choices made are so important to performance. Shopify engineers use a combination of Ruby on Rails, MySQL and Google products like Kubernetes. Resiliency systems like active-active configurations, chat ops for quick solutions, and bot and overload protection are worked in. By leveraging these tools and staying flexible in their resiliency efforts, Shopify is able to adjust to new merchant requirements and teams are able to work efficiently.

While tech continues to progress and change, the Shopify culture remains a driving force for advancement, Camilo tells us. The company ideals and axioms help steer the brand and dictate which technologies they'll use to solve new and changing client demands. The 2014 outage shaped the future of these cultural ideals, emphasizing the need for quick action and resiliency components like constraints to ensure system safety. Shopify engineers also built enhanced testing tools like Toxiproxy to simulate poor network conditions and account for potential issues.

The 2021 Black Friday Cyber Monday shopping season was Shopify's biggest yet. Camilo and Tai describe how Shopify's resiliency culture and intense prep work made the biggest shopping weekend of the year so successful. By offering educational tools and a support network that values good communication, their company culture continues to grow, and Tai tells us how it's not just the software that should be resilient. Building a resilient, flexible company culture is just as important.

Camilo talks about Shopify's recent shift to a completely remote work place and the new challenges and opportunities it presents.

Camilo Lopez

Camilo has worked at Shopify for more than 10 years, he has been an IC and a manager leading teams that take care of Shopify's scalability and resiliency.

Tai Dickerson

Tai is a production engineer at Shopify, where she shares her passion for resilience engineering with others via paper discussions and as a leader in Shopify's Resiliency SIG.

Cool things of the week

• Machine images is GA docs
• New Cloud Logging and Monitoring capabilities
• • Monitoring third-party applications: MariaDB docs
  • Monitoring third-party applications: MySQL docs
  • Monitoring third-party applications: Memcached docs
  • Starting with version 2.8.0, the Ops Agent supports Ubuntu 21.10. For more information, see Linux operating systems docs

Interview

• Shopify site
• Kubernetes site
• GKE site
• Kafka site
• Redis site
• Elastic Search site
• Memcached docs
• Toxiproxy site
• Shopify Engineering site
• Shopify Careers site
• BFCM Twitter Thread site
• Shopify engineers deliver on peak performance during Black Friday Cyber Monday 2021 blog
• Cloud, Load, and Modular Code: What 2022 Looks Like for Shopify blog
• Terri Haber on Resiliency at Scale site
• Terri Haber on Enforced Pacing site
• Bart Jedrocha on Load Testing site
• Bart Jedrocha on Tooling for Load Testing site
• Bart Jedrocha on The Future of Load Testing site
• Ryan McIlmoyl on Code Red site
• Ryan McIlmoyl on Working with IMOC site
• Camilo Lopez on The 2014 Outage site
• Camilo Lopez on Holiday Season Learnings site
• Tai Dickerson on Doing Things Differently site
• Tai Dickerson on Learning & Community site

What's something cool you're working on?

Stephanie is working on season 2 of the Where the Internet Lives podcast.

Carter is working on season 2 of VM End to End.

Hosts

Carter Morgan and Stephanie Wong