Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Avsnitt(243)

What's New in Kubernetes Storage (DoK Day EU 2022) // Xing Yang

What's New in Kubernetes Storage (DoK Day EU 2022) // Xing Yang

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Kubernetes SIG Storage is responsible for ensuring storage is available for containers in...

28 Maj 20229min

What we've learned from running a PostgreSQL managed service on Kubernetes (DoK Day EU 2022) // Oleksii Kliukin

What we've learned from running a PostgreSQL managed service on Kubernetes (DoK Day EU 2022) // Oleksii Kliukin

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Kubernetes is an emerging platform of choice for deploying and running PostgresSQL. Deplo...

28 Maj 202211min

Weathering The Cloud Storm- Modern Data Management Patterns for Reliability and Availability (DoK Day EU 2022) // Denis Magda

Weathering The Cloud Storm- Modern Data Management Patterns for Reliability and Availability (DoK Day EU 2022) // Denis Magda

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) “Zero downtime” and “always-on” are illusions. All systems fail sooner or later, whether ...

28 Maj 202210min

Using Kubernetes to deliver a “serverless” service (DoK Day EU 2022) // Jim Walker

Using Kubernetes to deliver a “serverless” service (DoK Day EU 2022) // Jim Walker

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Serverless promises to change the way we consume software. It allows us to potentially pa...

28 Maj 202220min

The many uses of Kubernetes cross cluster migration of persistent data (DoK Day EU 2022) // Ryan Kaw

The many uses of Kubernetes cross cluster migration of persistent data (DoK Day EU 2022) // Ryan Kaw

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Multiple clusters exist in most Kubernetes environments today, and number of clusters wil...

28 Maj 20227min

The future of data on Kubernetes with Adobe and CNCF (DoK Day EU 2022) // Joseph Sandoval, Xing Yang & Sylvain Kalache

The future of data on Kubernetes with Adobe and CNCF (DoK Day EU 2022) // Joseph Sandoval, Xing Yang & Sylvain Kalache

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Some data-intensive workloads are easier to run in Kubernetes than others. Why? What need...

28 Maj 202217min

The Data on Kubernetes Landscape (DoK Day EU 2022) // Melissa Logan & Sylvain Kalache

The Data on Kubernetes Landscape (DoK Day EU 2022) // Melissa Logan & Sylvain Kalache

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) We know from the first Data on Kubernetes Report that 90% of respondents believe Kubernet...

27 Maj 202210min

Testing the Mettle- Evaluating data solutions for large-scale production to check who stacks up (DoK Day EU 2022) // Dinesh Majrekar

Testing the Mettle- Evaluating data solutions for large-scale production to check who stacks up (DoK Day EU 2022) // Dinesh Majrekar

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) The state of the CNCF Storage options has exploded in the past few years, but if you had ...

27 Maj 20229min

Populärt inom Teknik

uppgang-och-fall
bilar-med-sladd
elbilsveckan
market-makers
rss-elektrikerpodden
skogsforum-podcast
rss-technokratin
rss-laddstationen-med-elbilen-i-sverige
rss-veckans-ai
rss-uppgang-och-fall
bli-saker-podden
developers-mer-an-bara-kod
rss-en-ai-till-kaffet
rss-powerboat-sverige-podcast
natets-morka-sida
rss-fabriken-2
har-vi-akt-till-mars-an
hej-bruksbil
rss-milpodden
rss-snacka-om-ai