NerdOut@Spotify8 Feb 2024

26: A Trillion Events

How did we learn to do event delivery at scale at Spotify? It’s been a journey. When you do something like tap the play button in the Spotify app, that’s an event. And getting that event data is fundamental to the Spotify experience. Without it, we wouldn’t be able to make music recommendations, pay artists fairly, or track down pesky, hard-to-find bugs. At the most basic level, this seems like a straightforward process: record an event, send that event data to a server somewhere, do something useful with it. Easy, right? But now, multiply that process by 50 million events per second. So, how do we make sure all that important data is delivered reliably, from our client apps to the cloud?

Host and principal engineer Dave Zolotusky talks with 9-year Spotify veteran Riccardo Petrocco about our journey building a event delivery system that can reliably handle a trillion events around the world, moving from Kafka to the cloud, building systems that are simple enough so that nobody tries to find a way around them and encourages “doing the right thing”, the definition of “quality data”, the value of moving up the stack and focusing less on the data pipes and more on what’s in them, and how Backstage makes it easier for our developers to discover, consume, produce, and manage data.

Learn more about Spotify’s data journey:

Read what else we’re nerding out about on the Spotify Engineering Blog: engineering.atspotify.com

You should follow us on Twitter @SpotifyEng, LinkedIn, and YouTube!

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(37)

34: Redesigning our Apple TV App Release Notes

What does it take to ship a video-first Spotify experience on the biggest screen in your house?In this Release Notes episode of the NerdOut@Spotify podcast, you’ll hear about the redesigned Spotify ap...

5 Mar 21min

33: Ship Happens (When Agents Code)

At Spotify, we're doubling down on background coding agents. The goal is to scale code migrations and boost developer productivity beyond our existing automation system. Built on top of Fleet Manageme...

15 Jan 49min

32: ARM-ing Ourselves with Google Axion, Part 2

In part two of ARM-ing ourselves with Google Axion, Dave is joined by Max Charas, a senior staff engineer at Spotify who’s been leading much of the migration work on our Google Axion journey, for a de...

19 Nov 202536min

31: ARM-ing Ourselves with Google Axion, Part 1

Spotify is in the middle of a big shift in modern computing: moving from x86 processors to Google’s new ARM-based Axion chips. This isn’t just a hardware swap — it’s a disruptive change with ripple ef...

8 Okt 202531min

30: Building AiKA: Spotify’s AI Knowledge Assistant

Hear about the journey behind building Spotify’s AI knowledge assistant, aka AiKA — from hack projects to internal chatbot, to enterprise developer tool.While general purpose chatbots are being used e...

14 Aug 202538min

29: Deploying Our New Typeface: Spotify Mix

Last year Spotify launched a big update to the app: a new typeface. For most of us, changing fonts is easy. It’s just a dropdown menu away. But creating a whole new typeface and then rolling it out ac...

3 Apr 202532min

28: The CNCF Turns 10

Ever heard of Kubernetes? Envoy? Prometheus? You probably have. But what you may not know is that these projects are managed and sustained by the Cloud Native Computing Foundation, aka the CNCF. Cloud...

6 Mar 202539min

Join us for Wizards, Portals, and Spotify for Backstage

Register for Spotify’s roadmap webinar on April 30, 2024 — and see what’s coming next from Spotify for Backstage, the open source platform for building internal developer portals. We’ll show you our l...

19 Apr 20243min