Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal
Data on Kubernetes Community27 Mai 2022

Datashim - a framework for declarative management of datasets on Kubernetes (DoK Day EU 2022) // Srikumar Venugopal

https://go.dok.community/slack

https://dok.community/

From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)


Many ML pipelines depend on shared filesystems for input, output and intermediate data storage. Standards such as CSI have made it possible for applications in Kubernetes to access a variety of data storage systems. Yet, data scientists still have to deal with low-level details of data access in order to execute their pipelines in Kubernetes. Datashim is a framework that manages the lifecycle of a Dataset object, a CustomResourceDefinition that represents a source of data. Datashim takes care of the details of data access while Kubernetes pods can declaratively access the data by referencing a Dataset in their specifications. This talk will describe Datashim and the Dataset object, discuss its use in ML pipelines, and demonstrate how its pluggable architecture is designed for the development of caching, scheduling and governance plugins. Datashim is an incubating project of the Linux Foundation Data and AI Foundation


Srikumar Venugopal is a Research Scientist in IBM Research Europe in Dublin, Ireland. His research interests lie in the area of cloud computing and large-scale distributed systems, specifically in the topics of middleware, resource management, and scalability. He is the co-founder and current lead for the Datashim project.

Episoder(243)

DoK Specials - Learn by doing in the DoK Community // Bart Farrell

DoK Specials - Learn by doing in the DoK Community // Bart Farrell

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK As a community we are committed to making learning how to run stateful workloads on Kubernetes as accessible and inclusi...

14 Jun 202215min

DoK Talks #135 - DoK isn't just Database on Kubernetes // Patrick McFadin

DoK Talks #135 - DoK isn't just Database on Kubernetes // Patrick McFadin

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK What about your streaming and analytic workloads? If you are all-in on Kubernetes you can't forget about these important parts...

10 Jun 202246min

DoK Talks #134 - Introducing CloudNativePG // Gabriele Bartolini & Leonardo Cecchi

DoK Talks #134 - Introducing CloudNativePG // Gabriele Bartolini & Leonardo Cecchi

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK CloudNativePG is an open source operator for the orchestration of Postgres workloads with a primary and an arbitrary numb...

9 Jun 20221h 5min

Dok Talks #133 - My First 90 days with Clickhouse // Alkin Tezuysal

Dok Talks #133 - My First 90 days with Clickhouse // Alkin Tezuysal

https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This talk will tell the story of an analytics use case database from a non-OLAP and ACID-compliant RDBMS (MySQL) perspective. ...

8 Jun 202247min

DoK Specials - DEI Panel - We can do better

DoK Specials - DEI Panel - We can do better

https://go.dok.community/slack https://dok.community/ With: Melissa Logan - Director, Data on Kubernetes Lisa-Marie Namphy - Head of Developer Relations, Cockroach Labs Alexandra Rowell - Communi...

3 Jun 202257min

DoK Talks #132 - Time-series on SQL Server on Kubernetes on ARM64… without SQL Server! // Álvaro Hernández

DoK Talks #132 - Time-series on SQL Server on Kubernetes on ARM64… without SQL Server! // Álvaro Hernández

https://go.dok.community/slack https://dok.community/ ABSTRACT OF THE TALK Oh wow. What a weird title. Full of terms that don’t fit together. Or do they? This talk is for believers, those who belie...

2 Jun 20221h 5min

Why we created one more Operator for MySQL (DoK Day EU 2022) // Sergey Pronin

Why we created one more Operator for MySQL (DoK Day EU 2022) // Sergey Pronin

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) There are plenty Kubernetes Operators for MySQL, including our own at Percona. In this se...

28 Mai 20229min

Why run Postgres in Kubernetes (DoK Day EU 2022) // Gabriele Bartolini

Why run Postgres in Kubernetes (DoK Day EU 2022) // Gabriele Bartolini

https://go.dok.community/slack https://dok.community/ From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE) Postgres should run inside your Kubernetes cluster. Yes, inside, not outside Kubernetes. ...

28 Mai 202210min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
teknisk-sett
tomprat-med-gunnar-tjomlid
energi-og-klima
rss-impressions-2
shifter
nasjonal-sikkerhetsmyndighet-nsm
fornybaren
elektropodden
pedagogisk-intelligens
teknologi-og-mennesker
rss-ki-praten
smart-forklart
rss-for-alarmen-gar
rss-ai-forklart
rss-digitaliseringspadden
rss-ki-til-kaffen
rss-heis
kortslutning