Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal
Data on Kubernetes Community27 Mai 2022

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Episoder(243)

Operating FoundationDB on Kubernetes (DoK Day EU 2022) // Johannes M. Scheuermann

Operating FoundationDB on Kubernetes (DoK Day EU 2022) // Johannes M. Scheuermann

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) FoundationDB is an open-source distributed transactional Key-Value store that is used by ...

27 Mai 20228min

One Click to Run Apache Spark as a Service on Kubernetes (DoK Day EU 2022) // Bo Yang

One Click to Run Apache Spark as a Service on Kubernetes (DoK Day EU 2022) // Bo Yang

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) It is still challenging to run Apache Spark and other big data processing workload on Kub...

27 Mai 20229min

Microservices and Kubernetes for your Full Data Lifecycle (DoK Day EU 2022) // Steve Pousty

Microservices and Kubernetes for your Full Data Lifecycle (DoK Day EU 2022) // Steve Pousty

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Data doesn’t magically appear in our data centers. There are usually several phases and s...

27 Mai 202214min

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Developers (DoK Day EU 2022) // Arsh Sharma, Lapo Elisacci & Ramiro Berrelleza

Leveraging Running Stateful Workloads on Kubernetes for the Benefit of Developers (DoK Day EU 2022) // Arsh Sharma, Lapo Elisacci & Ramiro Berrelleza

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Kubernetes comes with a lot of useful features like Volumes and StatefulSets, which make ...

27 Mai 202214min

Kanister & Kopia - An Open-Source Data Protection Match Made in Heaven (DoK Day EU 2022) // Pavan Navarathna

Kanister & Kopia - An Open-Source Data Protection Match Made in Heaven (DoK Day EU 2022) // Pavan Navarathna

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Cloud-native applications comprise various components, including data services, storage s...

27 Mai 202213min

Is your database in Kubernetes production ready (DoK Day EU 2022) // Mykola Marzhan

Is your database in Kubernetes production ready (DoK Day EU 2022) // Mykola Marzhan

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) It only looks simple to run databases in Kubernetes. In fact, it is too many things neede...

27 Mai 202215min

How to protect your data (DoK Day EU 2022) // Sarah Julia Kriesch

How to protect your data (DoK Day EU 2022) // Sarah Julia Kriesch

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) How can you keep your data secure and how can you transfer them on a secure way? You will...

27 Mai 20226min

Growing up fast - Kubernetes and Real-Time Analytic Applications (DoK Day EU 2022) // Robert Hodges

Growing up fast - Kubernetes and Real-Time Analytic Applications (DoK Day EU 2022) // Robert Hodges

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Kubernetes is turning into a preferred platform for real-time analytic app that crunch bi...

27 Mai 202215min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
teknisk-sett
tomprat-med-gunnar-tjomlid
energi-og-klima
elektropodden
rss-impressions-2
nasjonal-sikkerhetsmyndighet-nsm
fornybaren
shifter
pedagogisk-intelligens
teknologi-og-mennesker
rss-for-alarmen-gar
rss-ai-forklart
rss-ki-praten
rss-polypod
rss-digitaliseringspadden
rss-ki-til-kaffen
smart-forklart
blaskjerm-brodrene