Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models
Training Data30 Jul 2024

Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models

As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem. Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode: Llama 3.1 405B paper Open Source AI Is the Way Forward: Mark Zuckerberg essay released with Llama 3.1. Mistral Large 2 The Bitter Lesson by Rich Sutton 00:00 Introduction 01:28 The Llama 3.1 405B launch 05:02 The open source license 07:01 What's in it for Meta? 10:19 Why not open source? 11:16 Will frontier models commoditize? 12:41 What about startups? 16:29 The Mistral team 19:36 Are all frontier strategies comparable? 22:38 Is model development becoming more like software development? 26:34 Agentic reasoning 29:09 What future levers will unlock reasoning? 31:20 Will coding and math lead to unlocks? 33:09 Small models 34:08 7X more data 37:36 Are we going to hit a wall? 39:49 Lightning round

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(109)

Box's Aaron Levie: On Reinventing Yourself in the AI Age and Enterprise Diffusion

Box's Aaron Levie: On Reinventing Yourself in the AI Age and Enterprise Diffusion

Starting a company is hard. Reinventing your company for AI as a public company with quarterly earnings results is even harder. Aaron Levie has pulled off the transition with Box and offers hard-won a...

15 Sep 1h 5min

Making Cities Awesome: Peregrine’s Nick Noone & Ben Rudolph

Making Cities Awesome: Peregrine’s Nick Noone & Ben Rudolph

Most public safety technology companies grow by collecting more data. Peregrine inverted the model: no sensors, no new data, a business built on connecting the data and information cities already own....

1 Sep 52min

Parallel’s Parag Agrawal: Building a New Web for AI Agents

Parallel’s Parag Agrawal: Building a New Web for AI Agents

Parag Agrawal is making a bet that goes against two decades of web search: agents will query the web a thousand times more than humans ever have, and the infrastructure built around human clicks is wr...

25 Aug 55min

Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again

Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again

Rich Sutton, who helped pioneer reinforcement learning and wrote the seminal AI essay The Bitter Lesson, has now cofounded Oak Lab with his former student Khurram Javed. Their goal: to build agents th...

18 Aug 53min

Chai Discovery's Bitter Lesson: Drug Design Is Another Scaling Problem

Chai Discovery's Bitter Lesson: Drug Design Is Another Scaling Problem

Most people treat biology as a bespoke, messy science. Josh Meier and Matt McPartlon, co-founders of Chai Discovery, treat it as an engineering problem. They make the case that drug design obeys the b...

4 Aug 47min

Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Jerry Tworek led reasoning at OpenAI, convinced that scaling reinforcement learning was the path to AGI. Rohan Anil co-led Gemini pre-training and built the Shampoo optimizer. Now they've teamed up at...

29 Jul 49min

Factory's Matan Grinberg: The Coming ‘Dark Factory’ Where Software Builds Itself

Factory's Matan Grinberg: The Coming ‘Dark Factory’ Where Software Builds Itself

Factory started building fully autonomous coding agents in April 2023, two years before enterprises were ready. Matan Grinberg now says this is indistinguishable from being wrong. The Factory co-found...

21 Jul 51min

Anthropic's Katelyn Lesse & Angela Jiang: Building an Ecosystem, not a Walled Garden

Anthropic's Katelyn Lesse & Angela Jiang: Building an Ecosystem, not a Walled Garden

Katelyn Lesse and Angela Jiang lead the team building Anthropic's developer platform - the layer that both outside builders and Anthropic's own products run on top of. Angela frames the platform as a ...

14 Jul 48min

Populært innen Business og økonomi

stopp-verden
dine-penger-pengeradet
rss-penger-polser-og-politikk
e24-podden
rss-borsmorgen-okonominyhetene
rss-pa-konto
rss-skravla-gar
pengepodden-2
lederpodden
livet-pa-veien-med-jan-erik-larssen
tid-er-penger-en-podcast-med-peter-warren
stormkast-med-valebrokk-stordalen
pengesnakk
utbytte
rss-orjasater
morgenkaffen-med-finansavisen
finansredaksjonen
liberal-halvtime
rss-markedspuls-2
rss-fa-makro