Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Avsnitt(243)

Dok Specials - Ask Us Anything About Postgres // Gabriele Bartolini, Ryan Booz & Álvaro Hernández

Dok Specials - Ask Us Anything About Postgres // Gabriele Bartolini, Ryan Booz & Álvaro Hernández

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK What's the deal with Postgres in Kubernetes? To get some answers as well as more questions, we're bringing together Álvaro He...

23 Feb 20221h 3min

Dok Specials - Ask Patrick and Jeff Anything About Data on Kubernetes // Patrick McFadin & Jeff Carpenter

Dok Specials - Ask Patrick and Jeff Anything About Data on Kubernetes // Patrick McFadin & Jeff Carpenter

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Patrick is a Data on Kubernetes Community veteran. He did the very first session "Is k8s even ready for data?" in July 2020 a...

20 Feb 20221h 2min

Dok Specials - Unravel the key to your Kubernetes secrets

Dok Specials - Unravel the key to your Kubernetes secrets

https://go.dok.community/slack https://dok.community/

20 Feb 20222h 14min

Dok Talks #116 - Nebula Graph: Open Source Distributed Graph Database // Wey (Siwei) Gu

Dok Talks #116 - Nebula Graph: Open Source Distributed Graph Database // Wey (Siwei) Gu

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Nebula Graph Demystified Graph on K8s Know-How of Graph Database BIO Open Source believer, builder, singer and Graph Ma...

11 Feb 20221h 7min

Dok Special - Show me the money: The business side of DoK // Evan Powell, Brian Schechter & Misha Herscu

Dok Special - Show me the money: The business side of DoK // Evan Powell, Brian Schechter & Misha Herscu

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Running stateful workloads on Kubernetes isn't just a technical question. Without keeping the business value it provides in...

3 Feb 202257min

Dok Talks #115 - What More Can I Learn From My OpenTelemetry Traces? // John Pruitt

Dok Talks #115 - What More Can I Learn From My OpenTelemetry Traces? // John Pruitt

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Of the three observability data types supported by OpenTelemetry (metrics, logs, and traces) the latter is the one with mos...

2 Feb 20221h

Dok Talks #114 - Helm for Beginners with Portainer // Hrittik Roy

Dok Talks #114 - Helm for Beginners with Portainer // Hrittik Roy

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Helm allows you to install packages in your cluster, much like you would use apt, yum on your laptop. Just define the compo...

28 Jan 202251min

Dok Talks #113 - Developing Stateful Application on Kubernetes // Rob Pacheco

Dok Talks #113 - Developing Stateful Application on Kubernetes // Rob Pacheco

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Modern web applications are typically comprised of multiple services which utilize storage in a variety of ways. Utilizing ...

27 Jan 202253min

Populärt inom Teknik

uppgang-och-fall
elbilsveckan
market-makers
bilar-med-sladd
rss-elektrikerpodden
skogsforum-podcast
rss-laddstationen-med-elbilen-i-sverige
rss-technokratin
rss-uppgang-och-fall
rss-veckans-ai
natets-morka-sida
bli-saker-podden
developers-mer-an-bara-kod
rss-powerboat-sverige-podcast
rss-fabriken-2
har-vi-akt-till-mars-an
rss-en-ai-till-kaffet
rss-snacka-om-ai
hej-bruksbil
vi-bilagares-podcast