DoK Talks #101- Redpanda: how to build a storage engine for kubernetes // Alexander Gallego
Data on Kubernetes Community12 Marras 2021

DoK Talks #101- Redpanda: how to build a storage engine for kubernetes // Alexander Gallego

https://go.dok.community/slack
https://dok.community/

ABSTRACT OF THE TALK

How to build a stateful, transactional streaming storage engine on top of kubernetes.

KEY TAKE-AWAYS FROM THE TALK

1. Building storage engines in kubernetes is hard
2. You need tiered storage (S3 or google cloud bucket, Azure Blob) for total disaster recovery
3. You also need sound replication inside the cluster
4. You need a simple architecture so you can scale the pods easily

BIO

Alex Gallego is the founder and CEO of Vectorized, where he & the team hack on Redpanda, a modern streaming platform for mission critical workloads. Prior to Vectorized, he was a principal engineer at Akamai, as well as co-founder and CTO of Concord.io, a high performance stream processing engine built in C++ and acquired by Akamai in 2016.

Jaksot(243)

Graph in Kubernetes Panel (DoK Day EU 2022) // Wey Gu, Cheukting Ho & Feynman Zhou

Graph in Kubernetes Panel (DoK Day EU 2022) // Wey Gu, Cheukting Ho & Feynman Zhou

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Graph databases are the fastest growing data store in the world. According to Gartner, th...

27 Touko 202220min

From Laptop to Cloud. Developing Cloud-Native Applications with Containerized Databases (DoK Day EU 2022) - Nic Vermandé

From Laptop to Cloud. Developing Cloud-Native Applications with Containerized Databases (DoK Day EU 2022) - Nic Vermandé

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) With the advent of microservices in Kubernetes, individual developer teams now manage the...

27 Touko 202217min

Disaggregated Container Attached Storage - Yet Another Topology with What Purpose (DoK Day EU 2022) // Nick Connolly

Disaggregated Container Attached Storage - Yet Another Topology with What Purpose (DoK Day EU 2022) // Nick Connolly

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) The storage topology in vogue seems to cycle every few years. Internal storage is follow...

27 Touko 20229min

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Many ML pipelines depend on shared filesystems for input, output and intermediate data s...

27 Touko 202215min

Build your own social media analytics with Apache Kafka (DoK Day EU 2022) // Jakub Scholz

Build your own social media analytics with Apache Kafka (DoK Day EU 2022) // Jakub Scholz

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Apache Kafka is more than just a messaging broker. It has a rich ecosystem of different...

27 Touko 202210min

Bringing Apache Cassandra closer to Kubernetes (DoK Day EU 2022) // Jake Luciani

Bringing Apache Cassandra closer to Kubernetes (DoK Day EU 2022) // Jake Luciani

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) What does Kubernetes provide that allows us to reduce the complexity of Apache Cassandr...

27 Touko 20229min

Autoscaling Stateful Workloads in Kubernetes (DoK Day EU 2022) // Mohammad Fahim Abrar & Md. Kamol Hasan

Autoscaling Stateful Workloads in Kubernetes (DoK Day EU 2022) // Mohammad Fahim Abrar & Md. Kamol Hasan

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Managing stateful workloads in a containerized environment has always been a concern....

27 Touko 202210min

1000 node Cassandra cluster on Amazon's EKS (DoK Day EU 2022) // Matt Overstreet

1000 node Cassandra cluster on Amazon's EKS (DoK Day EU 2022) // Matt Overstreet

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Come here about our experience scaling Cassandra on EKS to over 1000 nodes and 20 million...

27 Touko 20227min