Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models
Training Data30 Heinä 2024

Meta’s Joe Spisak on Llama 3.1 405B and the Democratization of Frontier Models

As head of Product Management for Generative AI at Meta, Joe Spisak leads the team behind Llama, which just released the new 3.1 405B model. We spoke with Joe just two days after the model’s release to ask what’s new, what it enables, and how Meta sees the role of open source in the AI ecosystem. Joe shares that where Llama 3.1 405B really focused is on pushing scale (it was trained on 15 trillion tokens using 16,000 GPUs) and he’s excited about the zero-shot tool use it will enable, as well as its role in distillation and generating synthetic data to teach smaller models. He tells us why he thinks even frontier models will ultimately commoditize—and why that’s a good thing for the startup ecosystem. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode: Llama 3.1 405B paper Open Source AI Is the Way Forward: Mark Zuckerberg essay released with Llama 3.1. Mistral Large 2 The Bitter Lesson by Rich Sutton 00:00 Introduction 01:28 The Llama 3.1 405B launch 05:02 The open source license 07:01 What's in it for Meta? 10:19 Why not open source? 11:16 Will frontier models commoditize? 12:41 What about startups? 16:29 The Mistral team 19:36 Are all frontier strategies comparable? 22:38 Is model development becoming more like software development? 26:34 Agentic reasoning 29:09 What future levers will unlock reasoning? 31:20 Will coding and math lead to unlocks? 33:09 Small models 34:08 7X more data 37:36 Are we going to hit a wall? 39:49 Lightning round

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(105)

Chai Discovery's Bitter Lesson: Drug Design Is Another Scaling Problem

Chai Discovery's Bitter Lesson: Drug Design Is Another Scaling Problem

Most people treat biology as a bespoke, messy science. Josh Meier and Matt McPartlon, co-founders of Chai Discovery, treat it as an engineering problem. They make the case that drug design obeys the b...

4 Elo 47min

Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Jerry Tworek led reasoning at OpenAI, convinced that scaling reinforcement learning was the path to AGI. Rohan Anil co-led Gemini pre-training and built the Shampoo optimizer. Now they've teamed up at...

29 Heinä 49min

Factory's Matan Grinberg: The Coming ‘Dark Factory’ Where Software Builds Itself

Factory's Matan Grinberg: The Coming ‘Dark Factory’ Where Software Builds Itself

Factory started building fully autonomous coding agents in April 2023, two years before enterprises were ready. Matan Grinberg now says this is indistinguishable from being wrong. The Factory co-found...

21 Heinä 51min

Anthropic's Katelyn Lesse & Angela Jiang: Building an Ecosystem, not a Walled Garden

Anthropic's Katelyn Lesse & Angela Jiang: Building an Ecosystem, not a Walled Garden

Katelyn Lesse and Angela Jiang lead the team building Anthropic's developer platform - the layer that both outside builders and Anthropic's own products run on top of. Angela frames the platform as a ...

14 Heinä 48min

Inside Zipline's Autonomous System: 140M Miles, Zero Incidents

Inside Zipline's Autonomous System: 140M Miles, Zero Incidents

The largest commercial autonomous system on earth isn't a robotaxi fleet — it's Zipline, which has flown 140 million autonomous miles with zero safety incidents. Co-founder Keller Rinaudo Cliffton and...

7 Heinä 55min

Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Why Hardware-Software Co-Design Is AI's Real 100x: Dylan Patel of SemiAnalysis

Dylan Patel, founder of SemiAnalysis, argues the biggest gains in AI don't come from faster chips, they come from software-hardware co-design. Optimizing the model, the kernels, and the silicon togeth...

30 Kesä 1h 10min

Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

Memory and Continual Learning: Engram's Dan Biderman and Jessy Lin

Dan Biderman and Jessy Lin, co-founders of Engram, are building a neolab around memory and continual learning, which they call two sides of the same coin. Their contrarian premise: instead of stuffing...

24 Kesä 44min

Simulating Humans at Scale: Simile's Joon Sung Park

Simulating Humans at Scale: Simile's Joon Sung Park

The race to build superintelligence is producing models that keep getting better at objective problems, but not at behaving like actual people. Joon Sung Park, founder and CEO of Simile and creator of...

16 Kesä 38min

Suosittua kategoriassa Liike-elämä ja talous

sijotuskasti
psykopodiaa-podcast
mimmit-sijoittaa
ostan-asuntoja-podcast
rss-rahapodi
raharesepti
pomojen-suusta
pari-sanaa-lastensuojelusta
rss-sami-miettinen-neuvottelija
rss-porssipodi
rss-oivalluksia-rahasta-elamasta
rss-h-asselmoilanen
rss-laiskan-yrittajan-markkinointi-podcast
rss-myyntiradio
rss-deloitten-podcastit
the-mel-schmidt-show-yrittajyys-lifestyle-kasvu
pelkkana-korvana
rss-regenerative-business-uudistava-liiketoiminta
juristipodi
rss-lahtijat