#37 DoK Community: Running Data Replication Pipelines on Kubernetes with Argo // Stephen Bailey

#37 DoK Community: Running Data Replication Pipelines on Kubernetes with Argo // Stephen Bailey

Abstract of the talk…

Hundreds of data teams have migrated to the ELT pattern in recent years, leveraging SaaS tools like Stitch or FiveTran to reliably load data into their infrastructure. These SaaS offerings are outstanding and can accelerate your time to production significantly. However, many teams prefer to roll their own tools. One solution in these cases is to deploy singer.io taps and targets — Python scripts that can perform data replication between arbitrary sources and destinations. The Singer specification is the foundation for the popular Stitch SaaS, and it is also leveraged by a number of independent consultants and data projects. Singer pipelines are highly modular. You can pipe any tap to any target to build a data pipeline that fits your needs, making them a good fit for containerized workflows. This article walks through the workflow at a high level and provides some example code to get up and running with some shared templates. I also drill into reasons for choosing the Argo approach over other orchestration tools like Airflow or Dagster, and the implications from a team perspective.

Bio…

Stephen Bailey is Director of Growth Analytics at Immuta, where he strives to implement privacy best practices while delivering business value from data. He loves to teach and learn, on just about any subject. He holds a PhD in educational cognitive neuroscience from Vanderbilt and enjoys reading philosophy

Episoder(243)

DoK Specials - Learn by doing in the DoK Community // Bart Farrell

DoK Specials - Learn by doing in the DoK Community // Bart Farrell

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK As a community we are committed to making learning how to run stateful workloads on Kubernetes as accessible and inclusi...

14 Jun 202215min

DoK Talks #135 - DoK isn't just Database on Kubernetes // Patrick McFadin

DoK Talks #135 - DoK isn't just Database on Kubernetes // Patrick McFadin

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK What about your streaming and analytic workloads? If you are all-in on Kubernetes you can't forget about these important parts...

10 Jun 202246min

DoK Talks #134 - Introducing CloudNativePG // Gabriele Bartolini & Leonardo Cecchi

DoK Talks #134 - Introducing CloudNativePG // Gabriele Bartolini & Leonardo Cecchi

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK CloudNativePG is an open source operator for the orchestration of Postgres workloads with a primary and an arbitrary numb...

9 Jun 20221h 5min

Dok Talks #133 - My First 90 days with Clickhouse // Alkin Tezuysal

Dok Talks #133 - My First 90 days with Clickhouse // Alkin Tezuysal

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This talk will tell the story of an analytics use case database from a non-OLAP and ACID-compliant RDBMS (MySQL) perspective. ...

8 Jun 202247min

DoK Specials - DEI Panel - We can do better

DoK Specials - DEI Panel - We can do better

https://go.dok.community/slack https://dok.community/ With: Melissa Logan - Director, Data on Kubernetes Lisa-Marie Namphy - Head of Developer Relations, Cockroach Labs Alexandra Rowell - Communi...

3 Jun 202257min

DoK Talks #132 - Time-series on SQL Server on Kubernetes on ARM64… without SQL Server! // Álvaro Hernández

DoK Talks #132 - Time-series on SQL Server on Kubernetes on ARM64… without SQL Server! // Álvaro Hernández

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Oh wow. What a weird title. Full of terms that don’t fit together. Or do they? This talk is for believers, those who belie...

2 Jun 20221h 5min

Why we created one more Operator for MySQL (DoK Day EU 2022) // Sergey Pronin

Why we created one more Operator for MySQL (DoK Day EU 2022) // Sergey Pronin

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) There are plenty Kubernetes Operators for MySQL, including our own at Percona. In this se...

28 Mai 20229min

Why run Postgres in Kubernetes (DoK Day EU 2022) // Gabriele Bartolini

Why run Postgres in Kubernetes (DoK Day EU 2022) // Gabriele Bartolini

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Postgres should run inside your Kubernetes cluster. Yes, inside, not outside Kubernetes. ...

28 Mai 202210min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
teknisk-sett
tomprat-med-gunnar-tjomlid
rss-impressions-2
shifter
fornybaren
teknologi-og-mennesker
nasjonal-sikkerhetsmyndighet-nsm
energi-og-klima
elektropodden
rss-ki-praten
smart-forklart
pedagogisk-intelligens
rss-ai-forklart
rss-for-alarmen-gar
rss-heis
hans-petter-og-co
digital-forretningsforstaelse
rss-ki-til-kaffen