#044 - Data-efficient Image Transformers (Hugo Touvron)
Machine Learning Street Talk (MLST)25 Helmi 2021

#044 - Data-efficient Image Transformers (Hugo Touvron)

Today we are going to talk about the *Data-efficient image Transformers paper or (DeiT) which Hugo is the primary author of. One of the recipes of success for vision models since the DL revolution began has been the availability of large training sets. CNNs have been optimized for almost a decade now, including through extensive architecture search which is prone to overfitting. Motivated by the success of transformers-based models

in Natural Language Processing there has been increasing attention in applying these approaches to vision models. Hugo and his collaborators used a different training strategy and a new distillation token to get a massive increase in sample efficiency with image transformers.


00:00:00 Introduction

00:06:33 Data augmentation is all you need

00:09:53 Now the image patches are the convolutions though?

00:12:16 Where are those inductive biases hiding?

00:15:46 Distillation token

00:21:01 Why different resolutions on training

00:24:14 How data efficient can we get?

00:26:47 Out of domain generalisation

00:28:22 Why are transformers data efficient at all? Learning invariances

00:32:04 Is data augmentation cheating?

00:33:25 Distillation strategies - matching the intermediatae teacher representation as well as output

00:35:49 Do ML models learn the same thing for a problem?

00:39:01 How is it like at Facebook AI?

00:41:17 How long is the PhD programme?

00:42:03 Other interests outside of transformers?

00:43:18 Transformers for Vision and Language

00:47:40 Could we improve transformers models? (Hybrid models)

00:49:03 Biggest challenges in AI?

00:50:52 How far can we go with data driven approach?

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(260)

Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Tim Scarfe speaks with Ilia Shumailov and Alexander Panfilov about their paper, Stealing Reasoning Traces from Proprietary LLM APIs.The core bug sounds deceptively simple: providers return encrypted r...

22 Elo 49min

Every Exponential Ends — Silicon Valley Forgot — Adam Becker

Every Exponential Ends — Silicon Valley Forgot — Adam Becker

Astrophysicist Adam Becker, author of "What Is Real?", joins Tim Scarfe to take apart the futures Silicon Valley keeps selling: the 2045 singularity, mind uploading, Mars colonies, and the AI apocalyp...

20 Elo 1h 18min

AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyart

AI Is Learning at the Wrong Level of Abstraction — Matthieu Wyart

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstWhy can deep networks discover abstractions that shallow models miss? Statistical phys...

10 Elo 1h 18min

How Researchers Test AI for Hidden Goals — Apollo Research

How Researchers Test AI for Hidden Goals — Apollo Research

Can an AI do the right thing for the wrong reason? Tim Scarfe speaks with Apollo Research’s Alexander Meinke, Axel Højmark and Jérémy Scheurer about Measuring Reward-Seeking via Contrastive Belief Upd...

31 Heinä 1h 18min

Why a Nation Can't Outsource Its Frontier AI - Alistair Pullen (Cosine AI)

Why a Nation Can't Outsource Its Frontier AI - Alistair Pullen (Cosine AI)

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstBritain's most capable coding model can't be exported, and that ban is the whole reaso...

13 Heinä 55min

The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

The Benchmark With No Instructions — ARC-AGI-3 (winning team!)

Tim Scarfe travels to Zurich to sit down with the Tufa Labs ARC-AGI-3 team — founder Benjamin Crouzier, with Jeroen Cottaar, Dries Smit, Stefano Viel and Michal Tesnar — to work out what their leaderb...

1 Heinä 1h 24min

The Thermodynamic AI Computing Chip - Thomas Ahle

The Thermodynamic AI Computing Chip - Thomas Ahle

Thomas Ahle wants Normal Computing to be the Lovable for chip design: type your intent, and a swarm of agents carries it from design through optimisation, formalisation and verification to tape-out. T...

28 Kesä 1h 2min

He won a Nobel here for AlphaFold. Then he left. - John Jumper

He won a Nobel here for AlphaFold. Then he left. - John Jumper

This episode is sponsored by Notion. Learn more about Notion's Developer Platform today at https://notion.com/mlstProtein folding stalled biology for fifty years. A sequence of amino acids dictates a ...

22 Kesä 53min