Linear Digressions23 Feb 2020

The Lottery Ticket Hypothesis

Recent research into neural networks reveals that sometimes, not all parts of the neural net are equally responsible for the performance of the network overall. Instead, it seems like (in some neural nets, at least) there are smaller subnetworks present where most of the predictive power resides. The fascinating thing is that, for some of these subnetworks (so-called “winning lottery tickets”), it’s not the training process that makes them good at their classification or regression tasks: they just happened to be initialized in a way that was very effective. This changes the way we think about what training might be doing, in a pretty fundamental way. Sometimes, instead of crafting a good fit from wholecloth, training might be finding the parts of the network that always had predictive power to begin with, and isolating and strengthening them. This research is pretty recent, having only come to prominence in the last year, but nonetheless challenges our notions about what it means to train a machine learning model.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(331)

KV Cache Fundamentals

The unsung hero of every LLM conversation you've ever had? A clever optimization called the KV cache. Self-attention is computationally brutal — every token attending to every other token means costs ...

2 Nov 24min

AI Inference 101

Everyone talks about training the model — the data, the compute, the alignment magic. But what happens after the weights are frozen and millions of users start hammering it with prompts? That's infere...

26 Okt 27min

Back Next Week

Back Next Week by Katie Malone

19 Okt 18s

Back in Two Weeks

Back in Two Weeks by Katie Malone

12 Okt 21s

Mechanistic Interpretability: How Researchers Try To See Inside Models

As AI systems grow more powerful, they grow harder to understand — and that tradeoff has never mattered more. Most interpretability tricks give us approximations from the outside, but mechanistic inte...

5 Okt 29min

LLM As A Judge

Labeled data is expensive, slow, and painfully hard to come by — and when you're trying to evaluate whether an AI workflow is actually doing what you want, the problem gets even messier. How do you ju...

28 Sep 30min

The Impact of AI on Podcasting (Harvard Data Science Review Cross-Post)

Originally aired on the Harvard Data Science Review podcast. What can podcasting teach us about AI — and what can AI teach us about the future of podcasting? Katie Malone (that's our host) joins Jon ...

21 Sep 29min

Better Know A Benchmark: ExploitGym

When OpenAI's frontier models were caught hacking Hugging Face's servers, most people assumed they were hunting for answer keys. The real story is stranger and more unsettling. Katie and Phoebe unpack...

14 Sep 32min