Dok Talks #111 - Scheduled Scaling with Dask and Argo Workflows
Data on Kubernetes Community19 Tammi 2022

Dok Talks #111 - Scheduled Scaling with Dask and Argo Workflows

https://go.dok.community/slack
https://dok.community/

ABSTRACT OF THE TALK

Complex computational workloads in Python are a common sight these days, especially in the context of processing large and complex datasets. Battle-hardened modules such as Numpy, Pandas, and Scikit-Learn can perform low-level tasks, while tools like Dask makes it easy to parallelize these workloads across distributed computational environments. Meanwhile, Argo Workflows offers a Kubernetes-native solution to provisioning cloud resources in Kubernetes and triggering workflows on a regular schedule. Being Kubernetes-native, Argo Workflows also meshes nicely with other Kubernetes tools. This talk discusses the combination of these two worlds by showcasing a set-up for Argo-managed workflows which schedule and automatically scale-out Dask-powered data pipelines in Python.

BIO

Former academic in the field of renewable energy simulation and energy systems analysis. Currently responsible for architecting and maintaining the cloud- and data strategy at ACCURE Battery Intelligence

KEY TAKE-AWAYS FROM THE TALK

Argo Workflows + Dask is a nice combination for data-processing pipelines. There are a a few "gotchyas" to be on the look-out for, but in nevertheless this is still a generally-applicable and powerful combination.

https://github.com/sevberg

Jaksot(243)

Highly Available Postgres Clusters In Kubernetes // John Long & Jonathan Gonzalez (DoK Day North America 2022)

Highly Available Postgres Clusters In Kubernetes // John Long & Jonathan Gonzalez (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT A practical session about running Highly Available PostgreSQL in Kubernetes. The primary objective will be to demonstr...

2 Marras 202215min

Inter-Cluster PostreSQL on Kubernetes // Julian Fischer (DoK Day North America 2022)

Inter-Cluster PostreSQL on Kubernetes // Julian Fischer (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT In this talk you’ll explore how to run a PostgreSQL cluster across multiple Kubernetes clusters. Learn what challenges ar...

2 Marras 202217min

Open Source Databases on Kubernetes- Best Practices // Peter Zaitsev (DoK Day North America 2022)

Open Source Databases on Kubernetes- Best Practices // Peter Zaitsev (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT So you’re looking to run your Open Source Database on Kubernetes. What best practices should you follow and what pitfall...

2 Marras 202216min

The Kubernetes Native Database // Jeffrey Carpenter (DoK Day North America 2022)

The Kubernetes Native Database // Jeffrey Carpenter (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT In the software industry we’re fond of terms that define major trends, like “cloud native”, “Kubernetes native” and “server...

2 Marras 202216min

Databases on Kubernetes: Why are they important? // With Bhavin Shah, Xing Yang, Gabriele Bartolini & Patrick McFadin (DoK Day North America 2022)

Databases on Kubernetes: Why are they important? // With Bhavin Shah, Xing Yang, Gabriele Bartolini & Patrick McFadin (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT Kubernetes has crossed the chasm, but what about stateful applications and databases? Join us for this panel discussion and...

2 Marras 202234min

Data streaming on Kubernetes // Yaniv Ben Hemo (DoK Day North America 2022)

Data streaming on Kubernetes // Yaniv Ben Hemo (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT I will cover what is the current data streaming on k8s landscape, why it is important, use cases, and what are the challeng...

2 Marras 202213min

Architecting Your First Event Driven Serverless Streaming Applications on K8 // Timothy Spann (DoK Day North America 2022)

Architecting Your First Event Driven Serverless Streaming Applications on K8 // Timothy Spann (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT Once you have built a topic in Apache Pulsar, you will quickly see the need to build event-driven applications. This can r...

2 Marras 202213min

Fybrik - A Kubernetes based platform for governed data use // Flora Gilboa-Solomon, Alexey Roytman, Maryna Strelchuk & Barry Hijkoop (DoK Day North America 2022)

Fybrik - A Kubernetes based platform for governed data use // Flora Gilboa-Solomon, Alexey Roytman, Maryna Strelchuk & Barry Hijkoop (DoK Day North America 2022)

From the DoK Day North America 2022 (https://youtu.be/YWTa-DiVljY) ABSTRACT Data is the foundation for business value. However, in many enterprises, it is spread across different data stores, public...

1 Marras 202220min