Running Kafka on Kubernetes, across three clouds at Adobe (DoK Day EU 2022) // Adi Muraru

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)

Adobe runs dozens of Kafka clusters spread across both public (AWS and Azure) and private clouds to power the Adobe Experience Platform message bus.

In this talk, we share the experience of running Kafka as a stateful Kubernetes workload in a cloud-native environment using the open-source Koperator and the tools we built to automate the whole lifecycle.

While running a stateful service like Kafka on Kubernetes may be intimidating at the first glance, we share our thought process, the tools, and the results that can make this a reality in any organization. The Kubernetes Operator pattern helped us automate all the operational aspects for the lifecycle of the cluster; abstract away the cloud specifics allowing us to focus on Kafka; achieve increased resilience and elasticity; implement automated Kafka rebalancing using CruiseControl, and harness all the metrics to implement an observable environment. We also plan to demo how these all come together.

Join us as we share our team's learnings on:
- Considerations for running a stateful service in a multi-tenant Kubernetes cluster:
- Pod sizing in terms of memory and CPU
- Scheduling
- Persistent Volume Claims (PVC) tuning
- Resiliency topics
- How to handle regular Ethos upgrades and maintenance
- How to bring custom monitoring solutions
- Advanced network topics such as using dedicated VIPs

If you or your team ever wanted to learn how to run your own stateful service in Kubernetes and/or see what's possible, come join us!

Computer Science Ph.D. and Principal Scientist at Adobe having experience in complex, large scale distributed systems.

My focus currently is on designing and building the message streaming platform for Adobe Experience Platform using Apache Kafka. The endeavor is unique as it spans both Amazon AWS and Microsoft Azure clouds and handles billions of messages a day.