Mamba, Mamba-2 and Post-Transformer Architectures for Generative AI with Albert Gu - #693

Mamba, Mamba-2 and Post-Transformer Architectures for Generative AI with Albert Gu - #693

Today, we're joined by Albert Gu, assistant professor at Carnegie Mellon University, to discuss his research on post-transformer architectures for multi-modal foundation models, with a focus on state-space models in general and Albert’s recent Mamba and Mamba-2 papers in particular. We dig into the efficiency of the attention mechanism and its limitations in handling high-resolution perceptual modalities, and the strengths and weaknesses of transformer architectures relative to alternatives for various tasks. We dig into the role of tokenization and patching in transformer pipelines, emphasizing how abstraction and semantic relationships between tokens underpin the model's effectiveness, and explore how this relates to the debate between handcrafted pipelines versus end-to-end architectures in machine learning. Additionally, we touch on the evolving landscape of hybrid models which incorporate elements of attention and state, the significance of state update mechanisms in model adaptability and learning efficiency, and the contribution and adoption of state-space models like Mamba and Mamba-2 in academia and industry. Lastly, Albert shares his vision for advancing foundation models across diverse modalities and applications. The complete show notes for this episode can be found at https://twimlai.com/go/693.

Episoder(778)

Topological Data Analysis with Gunnar Carlsson - TWiML Talk #53

Topological Data Analysis with Gunnar Carlsson - TWiML Talk #53

The show you’re about to hear is part of a series of shows recorded in San Francisco at the Artificial Intelligence Conference. My guest for this show is Gunnar Carlsson, professor emeritus of mathema...

3 Okt 201733min

Bayesian Optimization for Hyperparameter Tuning with Scott Clark - TWiML Talk #50

Bayesian Optimization for Hyperparameter Tuning with Scott Clark - TWiML Talk #50

As you all know, a few weeks ago, I spent some time in SF at the Artificial Intelligence Conference. While I was there, I had just enough time to sneak away and catch up with Scott Clark, Co-Founder a...

2 Okt 201747min

Symbolic and Sub-Symbolic Natural Language Processing with Jonathan Mugan - TWiML Talk #49

Symbolic and Sub-Symbolic Natural Language Processing with Jonathan Mugan - TWiML Talk #49

Like last week’s interview with Bruno Goncalves, this week’s interview was also recorded at the last O’Reilly AI Conference back in New York in June. Also like last week’s show, this week’s is also fo...

25 Sep 201743min

Word2Vec & Friends with Bruno Gonçalves - TWiML Talk #48

Word2Vec & Friends with Bruno Gonçalves - TWiML Talk #48

This week i'm bringing you an interview from Bruno Goncalves, a Moore-Sloan Data Science Fellow at NYU. As you’ll hear in the interview, Bruno is a longtime listener of the podcast. We were able to co...

19 Sep 201732min

Evolutionary Algorithms in Machine Learning with Risto Miikkulainen - TWiML Talk #47

Evolutionary Algorithms in Machine Learning with Risto Miikkulainen - TWiML Talk #47

My guest this week is Risto Miikkulainen, professor of computer science at UT-Austin and vice president of Research at Sentient Technologies. Risto came locked and loaded to discuss a topic that we've...

11 Sep 201758min

Agile Machine Learning with Jennifer Prendki - TWiML Talk #46

Agile Machine Learning with Jennifer Prendki - TWiML Talk #46

My guest this week is Jennifer Prendki. That name might sound familiar, as she was one of the great speakers from my Future of Data Summit back in May. At the time, Jennifer was senior data science ma...

5 Sep 201748min

LSTMs, Plus a Deep Learning History Lesson with Jürgen Schmidhuber - TWiML Talk #44

LSTMs, Plus a Deep Learning History Lesson with Jürgen Schmidhuber - TWiML Talk #44

This week we have a very special interview to share with you! Those of you who’ve been receiving my newsletter for a while might remember that while in Switzerland last month, I had the pleasure of in...

28 Aug 20171h 3min

Machine Teaching for Better Machine Learning with Mark Hammond - TWiML Talk #43

Machine Teaching for Better Machine Learning with Mark Hammond - TWiML Talk #43

Today’s show, which concludes the first season of the Industrial AI Series, features my interview with Bonsai co-founder and CEO Mark Hammond. I sat down with Mark at Bonsai HQ a few weeks ago and we ...

21 Aug 20171h 5min

Populært innen Politikk og nyheter

giver-og-gjengen-vg
aftenpodden
aftenpodden-usa
forklart
stopp-verden
popradet
det-store-bildet
dine-penger-pengeradet
rss-gukild-johaug
bt-dokumentar-2
lydartikler-fra-aftenposten
hanna-de-heldige
fotballpodden-2
nokon-ma-ga
e24-podden
frokostshowet-pa-p5
aftenbla-bla
rss-ness
rss-penger-polser-og-politikk
rss-dannet-uten-piano