Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Avsnitt(243)

Dok Talks #131 - How to win friends and influence businesses // Fabian Met

Dok Talks #131 - How to win friends and influence businesses // Fabian Met

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK In this talk I share my personal experience where when I was working for a client the company had a hard time innovating and d...

12 Maj 20221h

Dok Talks #130- Leaning on Kubernetes Portability to Manage Databases Anywhere // Robert Hodges

Dok Talks #130- Leaning on Kubernetes Portability to Manage Databases Anywhere // Robert Hodges

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK What if databases only ran in a single place? That would be useless. But it's what we get with most database-as-a-service offe...

4 Maj 20221h 4min

Dok Talks #129 - Databases Operations and the Cloud // Barak Nissim

Dok Talks #129 - Databases Operations and the Cloud // Barak Nissim

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This session walks through the basics of how data is represented in Kubernetes, from the grounds up and explores how databases...

2 Maj 202251min

Dok Talks #126- Automatically Instrument Kubernetes Apps with OpenTelemetry // James Blackwood-Sewell

Dok Talks #126- Automatically Instrument Kubernetes Apps with OpenTelemetry // James Blackwood-Sewell

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK The rise of Kubernetes has triggered an exponential growth of metric and trace data. This talk explores capturing and persisti...

27 Apr 20221h 3min

Dok Talks #128- Getting Started with the Kubernetes Secrets Store CSI Driver // Kim Schlesinger

Dok Talks #128- Getting Started with the Kubernetes Secrets Store CSI Driver // Kim Schlesinger

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK In Kubernetes, it can be difficult to keep application API keys, access tokens and passwords safe. There are several different...

22 Apr 202253min

Dok Talks #127 - Flux for Helm Users! // Scott Rigby

Dok Talks #127 - Flux for Helm Users! // Scott Rigby

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK Welcome Helm users! CNCF Flux has a best-in-class way to use Helm according to GitOps principles. For you, that means improved...

21 Apr 20221h 21min

Dok Talks #125- Mission and Vision of the Rap-God-Project // Abhijith Ganesh

Dok Talks #125- Mission and Vision of the Rap-God-Project // Abhijith Ganesh

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK Explanation of how the how-to-dok project evolved into rap-god-api BIO An aspiring tech enthusiast who's massively into Data S...

21 Apr 202218min

Dok Talks #124 - Intro to Druid on Kubernetes // Sergio Ferragut

Dok Talks #124 - Intro to Druid on Kubernetes // Sergio Ferragut

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This talk will provide a high-level overview of Kubernetes, Helm charts and how they can be used to deploy Apache Druid cluste...

8 Apr 202254min

Populärt inom Teknik

uppgang-och-fall
elbilsveckan
market-makers
bilar-med-sladd
rss-elektrikerpodden
skogsforum-podcast
rss-laddstationen-med-elbilen-i-sverige
rss-technokratin
rss-uppgang-och-fall
rss-veckans-ai
natets-morka-sida
bli-saker-podden
developers-mer-an-bara-kod
rss-powerboat-sverige-podcast
rss-fabriken-2
har-vi-akt-till-mars-an
rss-en-ai-till-kaffet
rss-snacka-om-ai
hej-bruksbil
vi-bilagares-podcast