Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal
Data on Kubernetes Community27 Mai 2022

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Episoder(243)

Graph in Kubernetes Panel (DoK Day EU 2022) // Wey Gu, Cheukting Ho & Feynman Zhou

Graph in Kubernetes Panel (DoK Day EU 2022) // Wey Gu, Cheukting Ho & Feynman Zhou

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Graph databases are the fastest growing data store in the world. According to Gartner, th...

27 Mai 202220min

From Laptop to Cloud. Developing Cloud-Native Applications with Containerized Databases (DoK Day EU 2022) - Nic Vermandé

From Laptop to Cloud. Developing Cloud-Native Applications with Containerized Databases (DoK Day EU 2022) - Nic Vermandé

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) With the advent of microservices in Kubernetes, individual developer teams now manage the...

27 Mai 202217min

Disaggregated Container Attached Storage - Yet Another Topology with What Purpose (DoK Day EU 2022) // Nick Connolly

Disaggregated Container Attached Storage - Yet Another Topology with What Purpose (DoK Day EU 2022) // Nick Connolly

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) The storage topology in vogue seems to cycle every few years. Internal storage is follow...

27 Mai 20229min

Build your own social media analytics with Apache Kafka (DoK Day EU 2022) // Jakub Scholz

Build your own social media analytics with Apache Kafka (DoK Day EU 2022) // Jakub Scholz

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Apache Kafka is more than just a messaging broker. It has a rich ecosystem of different...

27 Mai 202210min

Bringing Apache Cassandra closer to Kubernetes (DoK Day EU 2022) // Jake Luciani

Bringing Apache Cassandra closer to Kubernetes (DoK Day EU 2022) // Jake Luciani

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) What does Kubernetes provide that allows us to reduce the complexity of Apache Cassandr...

27 Mai 20229min

Autoscaling Stateful Workloads in Kubernetes (DoK Day EU 2022) // Mohammad Fahim Abrar & Md. Kamol Hasan

Autoscaling Stateful Workloads in Kubernetes (DoK Day EU 2022) // Mohammad Fahim Abrar & Md. Kamol Hasan

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Managing stateful workloads in a containerized environment has always been a concern....

27 Mai 202210min

1000 node Cassandra cluster on Amazon's EKS (DoK Day EU 2022) // Matt Overstreet

1000 node Cassandra cluster on Amazon's EKS (DoK Day EU 2022) // Matt Overstreet

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Come here about our experience scaling Cassandra on EKS to over 1000 nodes and 20 million...

27 Mai 20227min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
teknisk-sett
tomprat-med-gunnar-tjomlid
energi-og-klima
elektropodden
rss-impressions-2
nasjonal-sikkerhetsmyndighet-nsm
fornybaren
shifter
pedagogisk-intelligens
teknologi-og-mennesker
rss-for-alarmen-gar
rss-ai-forklart
rss-ki-praten
rss-polypod
rss-digitaliseringspadden
rss-ki-til-kaffen
smart-forklart
blaskjerm-brodrene