Mamba, Mamba-2 and Post-Transformer Architectures for Generative AI with Albert Gu - #693

Mamba, Mamba-2 and Post-Transformer Architectures for Generative AI with Albert Gu - #693

Today, we're joined by Albert Gu, assistant professor at Carnegie Mellon University, to discuss his research on post-transformer architectures for multi-modal foundation models, with a focus on state-space models in general and Albert’s recent Mamba and Mamba-2 papers in particular. We dig into the efficiency of the attention mechanism and its limitations in handling high-resolution perceptual modalities, and the strengths and weaknesses of transformer architectures relative to alternatives for various tasks. We dig into the role of tokenization and patching in transformer pipelines, emphasizing how abstraction and semantic relationships between tokens underpin the model's effectiveness, and explore how this relates to the debate between handcrafted pipelines versus end-to-end architectures in machine learning. Additionally, we touch on the evolving landscape of hybrid models which incorporate elements of attention and state, the significance of state update mechanisms in model adaptability and learning efficiency, and the contribution and adoption of state-space models like Mamba and Mamba-2 in academia and industry. Lastly, Albert shares his vision for advancing foundation models across diverse modalities and applications. The complete show notes for this episode can be found at https://twimlai.com/go/693.

Avsnitt(781)

Learning Transformer Programs with Dan Friedman - #667

Learning Transformer Programs with Dan Friedman - #667

Today, we continue our NeurIPS series with Dan Friedman, a PhD student in the Princeton NLP group. In our conversation, we explore his research on mechanistic interpretability for transformer models, ...

15 Jan 202438min

AI Trends 2024: Machine Learning & Deep Learning with Thomas Dietterich - #666

AI Trends 2024: Machine Learning & Deep Learning with Thomas Dietterich - #666

Today we continue our AI Trends 2024 series with a conversation with Thomas Dietterich, distinguished professor emeritus at Oregon State University. As you might expect, Large Language Models figured ...

8 Jan 20241h 5min

AI Trends 2024: Computer Vision with Naila Murray - #665

AI Trends 2024: Computer Vision with Naila Murray - #665

Today we kick off our AI Trends 2024 series with a conversation with Naila Murray, director of AI research at Meta. In our conversation with Naila, we dig into the latest trends and developments in th...

2 Jan 202452min

Are Vector DBs the Future Data Platform for AI? with Ed Anuff - #664

Are Vector DBs the Future Data Platform for AI? with Ed Anuff - #664

Today we’re joined by Ed Anuff, chief product officer at DataStax. In our conversation, we discuss Ed’s insights on RAG, vector databases, embedding models, and more. We dig into the underpinnings of ...

28 Dec 202348min

Quantizing Transformers by Helping Attention Heads Do Nothing with Markus Nagel - #663

Quantizing Transformers by Helping Attention Heads Do Nothing with Markus Nagel - #663

Today we’re joined by Markus Nagel, research scientist at Qualcomm AI Research, who helps us kick off our coverage of NeurIPS 2023. In our conversation with Markus, we cover his accepted papers at the...

26 Dec 202346min

Responsible AI in the Generative Era with Michael Kearns - #662

Responsible AI in the Generative Era with Michael Kearns - #662

Today we’re joined by Michael Kearns, professor in the Department of Computer and Information Science at the University of Pennsylvania and an Amazon scholar. In our conversation with Michael, we disc...

22 Dec 202336min

Edutainment for AI and AWS PartyRock with Mike Miller - #661

Edutainment for AI and AWS PartyRock with Mike Miller - #661

Today we’re joined by Mike Miller, director of product at AWS responsible for the company’s “edutainment” products. In our conversation with Mike, we explore AWS PartyRock, a no-code generative AI app...

18 Dec 202329min

Data, Systems and ML for Visual Understanding with Cody Coleman - #660

Data, Systems and ML for Visual Understanding with Cody Coleman - #660

Today we’re joined by Cody Coleman, co-founder and CEO of Coactive AI. In our conversation with Cody, we discuss how Coactive has leveraged modern data, systems, and machine learning techniques to del...

14 Dec 202338min

Populärt inom Politik & nyheter

svenska-fall
aftonbladet-krim
p3-krim
rss-krimstad
fordomspodden
rss-expressen-dok
flashback-forever
motiv
aftonbladet-daily
rss-sanning-konsekvens
spar
blenda-2
rss-vad-fan-hande
olyckan-inifran
svd-ledarredaktionen
rss-krimreportrarna
rss-frandfors-horna
dagens-eko
rss-flodet
kungligt