Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera
Presented by Matt Menzenski, Senior Software Engineering Manager, Payitgov

At PayIt, we’ve been deploying applications to Kubernetes almost since the beginning of the company. Our data workloads, however, have run instead in AWS Glue. This has worked well enough for the reporting use cases that have been the main focus of this team historically. However, at the beginning of 2022, the PayIt data team began building out a new data platform, and in the process, ran into a number of challenges with Glue. In this talk, I will share the difficulties that we encountered with building, deploying, and orchestrating ETL pipelines in AWS Glue, our decision process for moving those workloads into Kubernetes, and the ELT architecture that we’ve arrived at today. Related Links DoKC Website - https://dok.community/ DoKC Meetups - https://www.meetup.com/data-on-kubernetes-community/ Join Slack - https://join.slack.com/t/dokcommunity/shared_invite/zt-1vgv7ymz7-YtLFvZicrcLP9fS3o_r2_w

Avsnitt(243)

Dok Talks #151 - Analytics with Apache Superset and ClickHouse // Vijay Anand Ramakrishnan

Dok Talks #151 - Analytics with Apache Superset and ClickHouse // Vijay Anand Ramakrishnan

https://go.dok.community/slack https://dok.community With: Vijay Anand Ramakrishnan - Database Administrator, ChistaDATA Bart Farrell - Head of Community, Data on Kubernetes Community ABSTRACT OF TH...

23 Sep 202233min

Dok Talks #150 - Building a Simple Postgres Async Streaming Cluster // Julian Fischer

Dok Talks #150 - Building a Simple Postgres Async Streaming Cluster // Julian Fischer

https://go.dok.community/slack https://dok.community With: Julian Fischer - CEO, anynines GmbH Bart Farrell - Head of Community, Data on Kubernetes Community ABSTRACT OF THE TALK In this talk you wi...

23 Sep 20221h 4min

DoK Talks #149 - Overcoming challenges with protecting and migrating data in multi-cloud K8s environments // Sebastian Glab & Martin Phan

DoK Talks #149 - Overcoming challenges with protecting and migrating data in multi-cloud K8s environments // Sebastian Glab & Martin Phan

https://go.dok.community/slack https://dok.community/ With: Sebastian Glab - Cloud Architect, CloudCasa by Catalogic Martin Phan - Field CTO – Americas, CloudCasa by Catalogic Bart Farrell - Head...

16 Sep 202247min

DoK Talks #147 - Evaluating Cloud Native Storage Vendors // Dinesh Majrekar

DoK Talks #147 - Evaluating Cloud Native Storage Vendors // Dinesh Majrekar

https://go.dok.community/slack https://dok.community/ With: Dinesh Majrekar - CTO, Civo Bart Farrell - Head of Community, Data on Kubernetes Community ABSTRACT OF THE TALK In a continuation of ...

5 Sep 20221h

Dok Talks #146 - OpenFeature - Making feature flags a commodity // Oleg Nenashev

Dok Talks #146 - OpenFeature - Making feature flags a commodity // Oleg Nenashev

https://go.dok.community/slack https://dok.community/ With: Oleg Nenashev - Community Builder and Developer Advocate, Dynatrace Bart Farrell - Head of Community, Data on Kubernetes Community AB...

26 Aug 20221h 1min

DoK Talks #145 - Making Hard Things Easy is Hard // Kurt Rinehart

DoK Talks #145 - Making Hard Things Easy is Hard // Kurt Rinehart

https://go.dok.community/slack https://dok.community/ https://youtu.be/6eSWOUzCb4w With: Kurt Rinehart - Director of Information Engineering, Section Bart Farrell - Head of Community, Data on Kube...

19 Aug 202257min

DoK Talks #144 - We will Dok You! - The journey to adopt stateful workloads on k8s // Guy Menahem

DoK Talks #144 - We will Dok You! - The journey to adopt stateful workloads on k8s // Guy Menahem

https://go.dok.community/slack https://dok.community/ https://youtu.be/AjvwG53yLMY With: Guy Menahem - Solution Architect, Komodor Bart Farrell - Head of Community, Data on Kubernetes Community A...

18 Aug 20221h 6min

DoK Talks #142 - Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your Stateful Workload // Peter Schuurman

DoK Talks #142 - Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your Stateful Workload // Peter Schuurman

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK How do you make sure your Stateful Workloads remain available when your Kubernetes infrastructure updates? This talk wil...

18 Aug 202258min

Populärt inom Teknik

uppgang-och-fall
elbilsveckan
bilar-med-sladd
skogsforum-podcast
market-makers
rss-elektrikerpodden
rss-uppgang-och-fall
rss-powerboat-sverige-podcast
gubbar-som-tjotar-om-bilar
rss-veckans-ai
rss-technokratin
hej-bruksbil
har-vi-akt-till-mars-an
developers-mer-an-bara-kod
bli-saker-podden
rss-fabriken-2
rss-en-ai-till-kaffet
rss-laddstationen-med-elbilen-i-sverige
rss-snacka-om-ai
rss-digitala-influencer-podden