Building Kafka without Disks with Richie Artoul and Ryan Worl from WarpStream Labs
Software Huddle16 Tammi 2024

Building Kafka without Disks with Richie Artoul and Ryan Worl from WarpStream Labs

In this episode, We spoke with the founders of WarpStream Labs, Richard Artoul and Ryan Worl. WarpStream is a fascinating rethink of Kafka -- how could you simplify and improve the Kafka design by slightly tweaking your constraints? The result is very compelling -- a Kafka-compatible API that bypasses local disk by writing everything directly to S3. For the tradeoff of a slightly higher end-to-end latency, you can get a Kafka cluster that's much cheaper and way easier to operate. Richie and Ryan have been working on high-scale data systems for years and were the engineers behind Husky, Datadog's custom-built database for logs and metrics. In this episode, they walk us through their experience building WarpStream. They touch on all the hard parts of building your own system (including why it's gotten easier!), as well as some of the difficult problems they had to solve for full compatibility with existing Kafka client libraries. They also touch on using FoundationDB, their thoughts on S3 Express One Zone, and whether AWS's cross-AZ network costs are a scam. Lots of interesting thoughts here from a really sharp team.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(79)

AI and Proactive Reliability with Kolton Andrus

AI and Proactive Reliability with Kolton Andrus

Today we're talking with Kolton Andrus, the Founder and CEO of Gremlin, about what happens to reliability when AI is writing most of the code. Kolton helped build the Chaos Engineering practice of bot...

8 Huhti 55min

Making Data Agent Ready with Andre Elizondo

Making Data Agent Ready with Andre Elizondo

Today we are talking with Andre Elizondo, the Director of Innovation at Mezmo about their open source agentic harness for SREs called AURA. Mezmo got their start handling observability data at scale. ...

27 Maalis 51min

Exponential Engineers with Ashmeet Sidana

Exponential Engineers with Ashmeet Sidana

Today on the show, we have a special guest — Ashmeet Sidana, the founder of Engineering Capital. Ashmeet started his career as an engineer at some great companies like Hewlett-Packard and Silicon Grap...

9 Tammi 53min

Powered by Neurons with Ewelina Kurtys

Powered by Neurons with Ewelina Kurtys

Today we have Dr. Ewelina Kurtys on the show. Ewelina has a background in Neuroscience and is currently working at FinalSpark. FinalSpark is using live Neurons for computations instead of traditiona...

16 Syys 202542min

Lessons from Building AI Agents with Rafal Wilinski

Lessons from Building AI Agents with Rafal Wilinski

Today we're talking with one of our favorite engineers, Rafal Wilinski. Rafal has been on the cutting edge of AI development in the last few years as he has led AI teams at Zapier and Vendr. Rafal wa...

12 Elo 20251h 8min

Building a High-Ownership Engineering Culture with Matt Watson

Building a High-Ownership Engineering Culture with Matt Watson

If you’ve ever felt like engineering teams are stuck in execution mode—heads down, building what they’re told—then today’s episode is for you. We're talking about what it really takes to build high ow...

5 Elo 202551min

Building CI for the age of AI Agents with Aayush Shah

Building CI for the age of AI Agents with Aayush Shah

Today's episode is with Aayush Shah. Aayush is one of the co-founders of Blacksmith, which is a CI compute platform. Basically, Blacksmith will run your GitHub Actions jobs faster and with more visibi...

22 Heinä 20251h 4min

Valkey After the Fork: A Conversation with Madelyn Olson

Valkey After the Fork: A Conversation with Madelyn Olson

Today, we're talking Valkey, Redis, and all things caching. Our guest is Madelyn Olson, who is a principal engineer at AWS working on Elasticache and is one of the most well-known people in the cachin...

16 Heinä 20251h 21min