#77 - Vitaliy Chiley (Cerebras)
Machine Learning Street Talk (MLST)16 Kesä 2022

#77 - Vitaliy Chiley (Cerebras)

Vitaliy Chiley is a Machine Learning Research Engineer at the next-generation computing hardware company Cerebras Systems. We spoke about how DL workloads including sparse workloads can run faster on Cerebras hardware.


[00:00:00] Housekeeping

[00:01:08] Preamble

[00:01:50] Vitaliy Chiley Introduction

[00:03:11] Cerebrus architecture

[00:08:12] Memory management and FLOP utilisation

[00:18:01] Centralised vs decentralised compute architecture

[00:21:12] Sparsity

[00:23:47] Does Sparse NN imply Heterogeneous compute?

[00:29:21] Cost of distributed memory stores?

[00:31:01] Activation vs weight sparsity

[00:37:52] What constitutes a dead weight to be pruned?

[00:39:02] Is it still a saving if we have to choose between weight and activation sparsity?

[00:41:02] Cerebras is a cool place to work

[00:44:05] What is sparsity? Why do we need to start dense?

[00:46:36] Evolutionary algorithms on Cerebras?

[00:47:57] How can we start sparse? Google RIGL

[00:51:44] Inductive priors, why do we need them if we can start sparse?

[00:56:02] Why anthropomorphise inductive priors?

[01:02:13] Could Cerebras run a cyclic computational graph?

[01:03:16] Are NNs locality sensitive hashing tables?


References;

Rigging the Lottery: Making All Tickets Winners [RIGL]

https://arxiv.org/pdf/1911.11134.pdf


[D] DanNet, the CUDA CNN of Dan Ciresan in Jurgen Schmidhuber's team, won 4 image recognition challenges prior to AlexNet

https://www.reddit.com/r/MachineLearning/comments/dwnuwh/d_dannet_the_cuda_cnn_of_dan_ciresan_in_jurgen/


A Spline Theory of Deep Learning [Balestriero]

https://proceedings.mlr.press/v80/balestriero18b.html

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(260)

Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Tim Scarfe speaks with Ilia Shumailov and Alexander Panfilov about their paper, Stealing Reasoning Traces from Proprietary LLM APIs.The core bug sounds deceptively simple: providers return encrypted r...

22 Elo 49min

Every Exponential Ends — Silicon Valley Forgot — Adam Becker

Every Exponential Ends — Silicon Valley Forgot — Adam Becker

Astrophysicist Adam Becker, author of "What Is Real?", joins Tim Scarfe to take apart the futures Silicon Valley keeps selling: the 2045 singularity, mind uploading, Mars colonies, and the AI apocalyp...

20 Elo 1h 18min

AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyart

AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyart

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstWhy can deep networks discover abstractions that shallow models miss? Statistical phys...

10 Elo 1h 18min

How Researchers Test AI for Hidden Goals — Apollo Research

How Researchers Test AI for Hidden Goals — Apollo Research

Can an AI do the right thing for the wrong reason? Tim Scarfe speaks with Apollo Research’s Alexander Meinke, Axel Højmark and Jérémy Scheurer about Measuring Reward-Seeking via Contrastive Belief Upd...

31 Heinä 1h 18min

Why a Nation Can't Outsource Its Frontier AI - Alistair Pullen (Cosine AI)

Why a Nation Can't Outsource Its Frontier AI - Alistair Pullen (Cosine AI)

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstBritain's most capable coding model can't be exported, and that ban is the whole reaso...

13 Heinä 55min

The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

Tim Scarfe travels to Zurich to sit down with the Tufa Labs ARC-AGI-3 team — founder Benjamin Crouzier, with Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tesnar — to work out what their leaderb...

1 Heinä 1h 24min

The Thermodynamic AI Computing Chip - Thomas Ahle

The Thermodynamic AI Computing Chip - Thomas Ahle

Thomas Ahle wants Normal Computing to be the Lovable for chip design: type your intent, and a swarm of agents carries it from design through optimisation, formalisation and verification to tape-out. T...

28 Kesä 1h 2min

He won a Nobel here for AlphaFold. Then he left. - John Jumper

He won a Nobel here for AlphaFold. Then he left. - John Jumper

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstProtein folding stalled biology for fifty years. A sequence of amino acids dictates a ...

22 Kesä 53min