Dok Talks #111 - Scheduled Scaling with Dask and Argo Workflows
Data on Kubernetes Community19 Tammi 2022

Dok Talks #111 - Scheduled Scaling with Dask and Argo Workflows

https://go.dok.community/slack
https://dok.community/

ABSTRACT OF THE TALK

Complex computational workloads in Python are a common sight these days, especially in the context of processing large and complex datasets. Battle-hardened modules such as Numpy, Pandas, and Scikit-Learn can perform low-level tasks, while tools like Dask makes it easy to parallelize these workloads across distributed computational environments. Meanwhile, Argo Workflows offers a Kubernetes-native solution to provisioning cloud resources in Kubernetes and triggering workflows on a regular schedule. Being Kubernetes-native, Argo Workflows also meshes nicely with other Kubernetes tools. This talk discusses the combination of these two worlds by showcasing a set-up for Argo-managed workflows which schedule and automatically scale-out Dask-powered data pipelines in Python.

BIO

Former academic in the field of renewable energy simulation and energy systems analysis. Currently responsible for architecting and maintaining the cloud- and data strategy at ACCURE Battery Intelligence

KEY TAKE-AWAYS FROM THE TALK

Argo Workflows + Dask is a nice combination for data-processing pipelines. There are a a few "gotchyas" to be on the look-out for, but in nevertheless this is still a generally-applicable and powerful combination.

https://github.com/sevberg

Jaksot(243)

Implementing Data & Databases on K8s within the Dutch Government | DoKC Town Hall

Implementing Data & Databases on K8s within the Dutch Government | DoKC Town Hall

Implementing Data & Databases on K8s within the Dutch GovernmentPresented by Sebastiaan Mannem, Director at Mannem Solutions A small walkthrough of projects within the Dutch government running databas...

13 Helmi 202444min

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall

Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and HeraPresented by Matt Menzenski, Senior Software Engineering Manager, Payitgov At PayIt, we’ve been deploying app...

6 Helmi 202423min

Repel Boarders! How to find a Kubernetes operator that really protects your data | DoKC Town Hall

Repel Boarders! How to find a Kubernetes operator that really protects your data | DoKC Town Hall

Repel Boarders! How to find a Kubernetes operator that really protects your dataPresented by Robert Hodges, AltinityOperators are a godsend for managing data in Kubernetes. But how about protecting it...

30 Tammi 202419min

DoK + Apache Spark | DoKC Town Hall

DoK + Apache Spark | DoKC Town Hall

DoK + Apache SparkPresented by Holden Karau, Spark Committer and Open Source Engineer at NetflixIn this brief talk, Holden will cover some of the best practices from trying to deploy both small and la...

23 Tammi 202419min

DoK @ Comcast - Deliver Business Outcomes & Improved DevX with Data Services on K8s | DoKC Town Hall

DoK @ Comcast - Deliver Business Outcomes & Improved DevX with Data Services on K8s | DoKC Town Hall

DoK @ Comcast: Delivering Business Outcomes & Improved DevX with Data Services Running on Kubernetes Presented by Greg Otto, Executor Director, DevX Platforms & Charles Ju, Principal Engineer Transfor...

3 Tammi 202416min

DoK Talks - What is Kafka? The rise of one of the world's most used streaming data technologies // Abbey Russell

DoK Talks - What is Kafka? The rise of one of the world's most used streaming data technologies // Abbey Russell

Abbey Russell, PM at Cockroach Labs, shared the backstory on how and why Kafka was created. Along the way, you'll learn about - Who Franz Kafka was - Kafka's earliest use at Linkedin in 2010 -...

9 Maalis 202315min

DoK Talks - (almost)Everything you need to know about stateful cloud native network applications // W Watson

DoK Talks - (almost)Everything you need to know about stateful cloud native network applications // W Watson

https://go.dok.community/slack https://dok.community/ https://youtu.be/KjiK6eXYO34 DoK Talk with W Watson, Founder at Vulk Co-op

2 Maalis 202343min

The Outer Nerd #001 - Dungeons & Dragons - Why should you care? // Abhi Vaidyanatha, Fabian Met & Chase Christensen

The Outer Nerd #001 - Dungeons & Dragons - Why should you care? // Abhi Vaidyanatha, Fabian Met & Chase Christensen

https://dokcommunity.slack.com/ https://dok.community/ ABSTRACT OF THE TALK Fabian, Chris and Abhi will discuss their passion for roleplaying games, and what they can teach us about the power of ...

13 Joulu 202258min