Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Avsnitt(243)

DoK #53 Day Zero - Azure Kubernetes Service // Raj Balakrishnan

DoK #53 Day Zero - Azure Kubernetes Service // Raj Balakrishnan

Abstract of the talk… Are you new to azure kubernetes service and just want to see how the nuts and bolts come together ? This is the talk to be. Single slide and a end to end demo on how to run your ...

9 Juni 20211h

DoK #52 Enterprise-grade Kubernetes requirements // Haseeb Budhani

DoK #52 Enterprise-grade Kubernetes requirements // Haseeb Budhani

Abstract of the talk… We'll discuss best practices companies are adopting for enterprise-grade Kubernetes Management. Bio… Haseeb Budhani is the CEO of Rafay Systems, which he co-founded in late 2017....

5 Juni 20211h 1min

DoK #51 Promscale: Using Prometheus + Promscale + PostgreSQL to go from Observation to Understanding // Matvey Arye

DoK #51 Promscale: Using Prometheus + Promscale + PostgreSQL to go from Observation to Understanding // Matvey Arye

Abstract of the talk… Often when I talk about putting observability data into PostgreSQL people ask me: are you crazy? And yet this somewhat heretical view has the potential to unlock a lot of the pow...

29 Maj 20211h

DoK #49 Deployments vs StatefulSets vs Daemonsets // Ali Kahoo

DoK #49 Deployments vs StatefulSets vs Daemonsets // Ali Kahoo

Abstract of the talk… Kubernetes provides different resources for deploying applications, we will be looking at them and the differences between them and how can we persist data using each of them. Bi...

29 Maj 20211h 8min

DoK #50 Going Full Circle with Kafka // Ravi Trivedi

DoK #50 Going Full Circle with Kafka // Ravi Trivedi

Abstract of the talk… Tecton is building a data platform for machine learning. This talk shares some of the adventures and lessons learned while introducing Kafka into our data pipelines. Bio… Enginee...

20 Maj 20211h 2min

DoK #48 Airflow vs Argo - Battle Royale // Tim van de Keer

DoK #48 Airflow vs Argo - Battle Royale // Tim van de Keer

Abstract of the talk… We are going to be looking at and comparing Airflow (the established) versus Argo Workflows (The new kid on the block) and see how they measure up. What you would use each for, w...

20 Maj 202159min

#1 DoK Community in Hindi: "Pehle Kadam Data on Kubernetes Community mein! // Kunal Kushwaha

#1 DoK Community in Hindi: "Pehle Kadam Data on Kubernetes Community mein! // Kunal Kushwaha

Abstract of the talk… Kya hota hai Kubernetes? Shuruwat kahan se kare? Community ka hissa kaise bane? Kya aap ke mann mein bhi ye sawaal aate hain? Join kariye hume iss meetup mein jahan hum baat kare...

4 Maj 20211h 2min

DoK Community #47 FullStack OpenSource Observability using SigNoz // Ankit Nayan

DoK Community #47 FullStack OpenSource Observability using SigNoz // Ankit Nayan

Abstract of the talk… In the talk, we shall dive deep into the latest open-source tools like Prometheus and Jaeger and our journey in using them and ultimately building our own open-source observabili...

4 Maj 202155min

Populärt inom Teknik

uppgang-och-fall
elbilsveckan
market-makers
bilar-med-sladd
rss-elektrikerpodden
skogsforum-podcast
rss-laddstationen-med-elbilen-i-sverige
rss-technokratin
rss-uppgang-och-fall
rss-veckans-ai
natets-morka-sida
bli-saker-podden
developers-mer-an-bara-kod
rss-powerboat-sverige-podcast
rss-fabriken-2
har-vi-akt-till-mars-an
rss-en-ai-till-kaffet
rss-snacka-om-ai
hej-bruksbil
vi-bilagares-podcast