Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Avsnitt(243)

DoK Talks #144 - Mastering MongoDB on Kubernetes, the power of operators // Arek Borucki

DoK Talks #144 - Mastering MongoDB on Kubernetes, the power of operators // Arek Borucki

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK During my first talk for DoK community I want to walk you through the world of NoSQL database MongoDB and Kubernetes Ope...

26 Juli 20221h

DoK Specials - Why are Operators paramount to running stateful workloads on Kubernetes?

DoK Specials - Why are Operators paramount to running stateful workloads on Kubernetes?

In this panel with Sylvain Kalache, Head of Content at the DoK Community, drives a conversation featuring Nic Vermandé- Principal Developer Advocate at Ondat, Julian Fischer- CEO at anynines, and Serg...

20 Juli 202253min

DoK Talks #141 - Dossier: multi-tenant distributed Jupyter Notebooks // Iacoppo Colonnelli & Dario Tranchitella

DoK Talks #141 - Dossier: multi-tenant distributed Jupyter Notebooks // Iacoppo Colonnelli & Dario Tranchitella

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK When providing data analysis as a service, one must tackle several problems. Data privacy and protection by design are crucia...

15 Juli 20221h

DoK Talks #140 - Data protection of stateful environment // Timothy Dewin

DoK Talks #140 - Data protection of stateful environment // Timothy Dewin

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK More and more we see stateful workloads pop up in Kubernetes clusters. These workloads generate data that is unique and is eph...

28 Juni 202242min

DoK Talks #139 - Private DBaaS on Kubernetes // Sergey Pronin

DoK Talks #139 - Private DBaaS on Kubernetes // Sergey Pronin

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK Percona is committed to deliver solutions to run open source databases anywhere without lock in. As part of this commitment, w...

28 Juni 202253min

DoK Talks #138 - Build your own social media analytics with Apache Kafka // Jakub Scholz

DoK Talks #138 - Build your own social media analytics with Apache Kafka // Jakub Scholz

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK Apache Kafka is more than just a messaging broker. It has a rich ecosystem of different components. There are connectors for i...

24 Juni 202256min

DoK Talks #137 - How to build your own “Doordash” app // Yaniv Ben Hemo

DoK Talks #137 - How to build your own “Doordash” app // Yaniv Ben Hemo

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK The entire app is built in microservices, running on k8s pods and uses k8s-native message broker called memphis WORKSHOP...

23 Juni 202257min

DoK Talks #136 - Building a mesh for databases from scratch and why // Maxwell Miao

DoK Talks #136 - Building a mesh for databases from scratch and why // Maxwell Miao

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK In this talk, Maxwell is going to share his thoughts about Service Mesh and database operations, called Database Mesh, a...

15 Juni 202247min

Populärt inom Teknik

uppgang-och-fall
bilar-med-sladd
elbilsveckan
market-makers
rss-elektrikerpodden
skogsforum-podcast
rss-technokratin
rss-laddstationen-med-elbilen-i-sverige
rss-veckans-ai
rss-uppgang-och-fall
bli-saker-podden
developers-mer-an-bara-kod
rss-en-ai-till-kaffet
rss-powerboat-sverige-podcast
natets-morka-sida
rss-fabriken-2
har-vi-akt-till-mars-an
hej-bruksbil
rss-milpodden
rss-snacka-om-ai