129 - Transformers and Hierarchical Structure, with Shunyu Yao
NLP Highlights2 Juli 2021

129 - Transformers and Hierarchical Structure, with Shunyu Yao

In this episode, we talk to Shunyu Yao about recent insights into how transformers can represent hierarchical structure in language. Bounded-depth hierarchical structure is thought to be a key feature of natural languages, motivating Shunyu and his coauthors to show that transformers can efficiently represent bounded-depth Dyck languages, which can be thought of as a formal model of the structure of natural languages. We went on to discuss some of the intuitive ideas that emerge from the proofs, connections to RNNs, and insights about positional encodings that may have practical implications. More broadly, we also touched on the role of formal languages and other theoretical tools in modern NLP. Papers discussed in this episode: - Self-Attention Networks Can Process Bounded Hierarchical Languages (https://arxiv.org/abs/2105.11115) - Theoretical Limitations of Self-Attention in Neural Sequence Models (https://arxiv.org/abs/1906.06755) - RNNs can generate bounded hierarchical languages with optimal memory (https://arxiv.org/abs/2010.07515) - On the Practical Computational Power of Finite Precision RNNs for Language Recognition (https://arxiv.org/abs/1805.04908) Shunyu Yao's webpage: https://ysymyth.github.io/ The hosts for this episode are William Merrill and Matt Gardner.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(145)

Are LLMs safe?

Are LLMs safe?

Curious about the safety of LLMs? 🤔 Join us for an insightful new episode featuring Suchin Gururangan, Young Investigator at Allen Institute for Artificial Intelligence and Data Science Engineer at A...

29 Feb 202442min

"Imaginative AI" with Mohamed Elhoseiny

"Imaginative AI" with Mohamed Elhoseiny

This podcast episode features Dr. Mohamed Elhoseiny, a true luminary in the realm of computer vision with over a decade of groundbreaking research. As an Assistant Professor at KAUST, Dr. Elhoseiny's ...

8 Jan 202423min

142 - Science Of Science, with Kyle Lo

142 - Science Of Science, with Kyle Lo

Our first guest with this new format is Kyle Lo, the most senior lead scientist in the Semantic Scholar team at Allen Institute for AI (AI2), who kindly agreed to share his perspective on #Science of ...

28 Dec 202348min

141 - Building an open source LM, with Iz Beltagy and Dirk Groeneveld

141 - Building an open source LM, with Iz Beltagy and Dirk Groeneveld

In this special episode of NLP Highlights, we discussed building and open sourcing language models. What is the usual recipe for building large language models? What does it mean to open source them? ...

29 Juni 202329min

140 - Generative AI and Copyright, with Chris Callison-Burch

140 - Generative AI and Copyright, with Chris Callison-Burch

In this special episode, we chatted with Chris Callison-Burch about his testimony in the recent U.S. Congress Hearing on the Interoperability of AI and Copyright Law. We started by asking Chris about ...

6 Juni 202351min

139 - Coherent Long Story Generation, with Kevin Yang

139 - Coherent Long Story Generation, with Kevin Yang

How can we generate coherent long stories from language models? Ensuring that the generated story has long range consistency and that it conforms to a high level plan is typically challenging. In this...

24 Mars 202345min

138 - Compositional Generalization in Neural Networks, with Najoung Kim

138 - Compositional Generalization in Neural Networks, with Najoung Kim

Compositional generalization refers to the capability of models to generalize to out-of-distribution instances by composing information obtained from the training data. In this episode we chatted with...

20 Jan 202348min

137 - Nearest Neighbor Language Modeling and Machine Translation, with Urvashi Khandelwal

137 - Nearest Neighbor Language Modeling and Machine Translation, with Urvashi Khandelwal

We invited Urvashi Khandelwal, a research scientist at Google Brain to talk about nearest neighbor language and machine translation models. These models interpolate parametric (conditional) language m...

13 Jan 202335min

Populärt inom Vetenskap

dumma-manniskor
allt-du-velat-veta
p3-dystopia
kapitalet-en-podd-om-ekonomi
rss-vetenskapsradion
rss-ufobortom-rimligt-tvivel
bildningspodden
paranormalt-med-caroline-giertz
rss-vetenskapsradion-2
svd-nyhetsartiklar
rss-spraket
sexet
det-morka-psyket
medicinvetarna
vetenskapsradion
halsorevolutionen
rss-ronden
rss-personlighetspodden
4health-med-anna-sparre
hacka-livet