#37 DoK Community: Running Data Replication Pipelines on Kubernetes with Argo // Stephen Bailey
Data on Kubernetes Community25 Mar 2021

#37 DoK Community: Running Data Replication Pipelines on Kubernetes with Argo // Stephen Bailey

Abstract of the talk…

Hundreds of data teams have migrated to the ELT pattern in recent years, leveraging SaaS tools like Stitch or FiveTran to reliably load data into their infrastructure. These SaaS offerings are outstanding and can accelerate your time to production significantly. However, many teams prefer to roll their own tools. One solution in these cases is to deploy singer.io taps and targets — Python scripts that can perform data replication between arbitrary sources and destinations. The Singer specification is the foundation for the popular Stitch SaaS, and it is also leveraged by a number of independent consultants and data projects. Singer pipelines are highly modular. You can pipe any tap to any target to build a data pipeline that fits your needs, making them a good fit for containerized workflows. This article walks through the workflow at a high level and provides some example code to get up and running with some shared templates. I also drill into reasons for choosing the Argo approach over other orchestration tools like Airflow or Dagster, and the implications from a team perspective.

Bio…

Stephen Bailey is Director of Growth Analytics at Immuta, where he strives to implement privacy best practices while delivering business value from data. He loves to teach and learn, on just about any subject. He holds a PhD in educational cognitive neuroscience from Vanderbilt and enjoys reading philosophy

Episoder(243)

Implementing Data & Databases on K8s within the Dutch Government | DoKC Town Hall

Implementing Data & Databases on K8s within the Dutch Government | DoKC Town Hall

Implementing Data & Databases on K8s within the Dutch GovernmentPresented by Sebastiaan Mannem, Director at Mannem Solutions A small walkthrough of projects within the Dutch government running databas...

13 Feb 202444min

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and HeraPresented by Matt Menzenski, Senior Software Engineering Manager, Payitgov At PayIt, we’ve been deploying app...

6 Feb 202423min

Repel Boarders! How to find a Kubernetes operator that really protects your data | DoKC Town Hall

Repel Boarders! How to find a Kubernetes operator that really protects your data | DoKC Town Hall

Repel Boarders! How to find a Kubernetes operator that really protects your dataPresented by Robert Hodges, AltinityOperators are a godsend for managing data in Kubernetes. But how about protecting it...

30 Jan 202419min

DoK + Apache Spark | DoKC Town Hall

DoK + Apache Spark | DoKC Town Hall

DoK + Apache SparkPresented by Holden Karau, Spark Committer and Open Source Engineer at NetflixIn this brief talk, Holden will cover some of the best practices from trying to deploy both small and la...

23 Jan 202419min

DoK @ Comcast - Deliver Business Outcomes & Improved DevX with Data Services on K8s | DoKC Town Hall

DoK @ Comcast - Deliver Business Outcomes & Improved DevX with Data Services on K8s | DoKC Town Hall

DoK @ Comcast: Delivering Business Outcomes & Improved DevX with Data Services Running on Kubernetes Presented by Greg Otto, Executor Director, DevX Platforms & Charles Ju, Principal Engineer Transfor...

3 Jan 202416min

DoK Talks - What is Kafka? The rise of one of the world's most used streaming data technologies // Abbey Russell

DoK Talks - What is Kafka? The rise of one of the world's most used streaming data technologies // Abbey Russell

Abbey Russell, PM at Cockroach Labs, shared the backstory on how and why Kafka was created. Along the way, you'll learn about - Who Franz Kafka was - Kafka's earliest use at Linkedin in 2010 -...

9 Mar 202315min

DoK Talks - (almost)Everything you need to know about stateful cloud native network applications // W Watson

DoK Talks - (almost)Everything you need to know about stateful cloud native network applications // W Watson

https://go.dok.community/slack https://dok.community/ https://youtu.be/KjiK6eXYO34 DoK Talk with W Watson, Founder at Vulk Co-op

2 Mar 202343min

The Outer Nerd #001 - Dungeons & Dragons - Why should you care? // Abhi Vaidyanatha, Fabian Met & Chase Christensen

The Outer Nerd #001 - Dungeons & Dragons - Why should you care? // Abhi Vaidyanatha, Fabian Met & Chase Christensen

https://dokcommunity.slack.com/ https://dok.community/ ABSTRACT OF THE TALK Fabian, Chris and Abhi will discuss their passion for roleplaying games, and what they can teach us about the power of ...

13 Des 202258min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
teknisk-sett
tomprat-med-gunnar-tjomlid
rss-impressions-2
shifter
teknologi-og-mennesker
fornybaren
nasjonal-sikkerhetsmyndighet-nsm
rss-ki-praten
smart-forklart
elektropodden
pedagogisk-intelligens
rss-alt-vi-kan
rss-ai-forklart
hans-petter-og-co
rss-for-alarmen-gar
rss-heis
energi-og-klima
rss-polypod