Speculative Decoding and Efficient LLM Inference with Chris Lott - #717

Speculative Decoding and Efficient LLM Inference with Chris Lott - #717

Today, we're joined by Chris Lott, senior director of engineering at Qualcomm AI Research to discuss accelerating large language model inference. We explore the challenges presented by the LLM encoding and decoding (aka generation) and how these interact with various hardware constraints such as FLOPS, memory footprint and memory bandwidth to limit key inference metrics such as time-to-first-token, tokens per second, and tokens per joule. We then dig into a variety of techniques that can be used to accelerate inference such as KV compression, quantization, pruning, speculative decoding, and leveraging small language models (SLMs). We also discuss future directions for enabling on-device agentic experiences such as parallel generation and software tools like Qualcomm AI Orchestrator. The complete show notes for this episode can be found at https://twimlai.com/go/717.

Jaksot(782)

Siraj Raval - How to Build Confidence as an ML Developer - TWiML Talk #2

Siraj Raval - How to Build Confidence as an ML Developer - TWiML Talk #2

Siraj Raval is a machine learning hacker and teacher whose machine learning for hackers and fresh machine learning youtube series are fun, informative, high energy and practical ways to learn about a ...

21 Elo 201640min

This Week in ML & AI – 8/12/16: Another huge machine learning acquisition + AI in the Olympics

This Week in ML & AI – 8/12/16: Another huge machine learning acquisition + AI in the Olympics

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. This week we discuss Intel’s latest deep...

15 Elo 201623min

This Week in ML & AI – 8/5/16: Apple Acquires Turi, the DARPA Hacker-Bot Challenge and More

This Week in ML & AI – 8/5/16: Apple Acquires Turi, the DARPA Hacker-Bot Challenge and More

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. This week we look at Apple’s acquisition...

6 Elo 201624min

Clare Corthell - Open Source Data Science Masters, Hybrid AI, Algorithmic Ethics - TWiML Talk #1

Clare Corthell - Open Source Data Science Masters, Hybrid AI, Algorithmic Ethics - TWiML Talk #1

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. We try something new this week with an i...

31 Heinä 201647min

This Week in ML & AI - 7/22/16: ML to Optimize Datacenters, Crazy New GPU from NVIDIA, Faster RNNs

This Week in ML & AI - 7/22/16: ML to Optimize Datacenters, Crazy New GPU from NVIDIA, Faster RNNs

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. This week covers Google’s use of ML to c...

24 Heinä 201625min

This Week in ML & AI - 7/15/16: A Wingman AI for Pokémon Go and Wide & Deep Learning at Google

This Week in ML & AI - 7/15/16: A Wingman AI for Pokémon Go and Wide & Deep Learning at Google

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. This week's show features a conversation...

17 Heinä 201630min

This Week in ML & AI - 7/8/16: A BS Meter for AI, Retrieval Models for Chatbots & Predatory Robots

This Week in ML & AI - 7/8/16: A BS Meter for AI, Retrieval Models for Chatbots & Predatory Robots

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. This week's show covers the White House’...

10 Heinä 201629min

This Week in ML & AI - 7/1/16: Fatal Tesla Autopilot Crash, EU Outlawing Machine Learning & CVPR

This Week in ML & AI - 7/1/16: Fatal Tesla Autopilot Crash, EU Outlawing Machine Learning & CVPR

This Week in Machine Learning & AI brings you the week’s most interesting and important stories from the world of machine learning and artificial intelligence. This week's show covers the first fatal ...

3 Heinä 201635min

Suosittua kategoriassa Politiikka ja uutiset

uutiscast
aikalisa
ootsa-kuullut-tasta-2
rss-ootsa-kuullut-tasta
politiikan-puskaradio
tervo-halme
rss-podme-livebox
rss-vaalirankkurit-podcast
et-sa-noin-voi-sanoo-esittaa
the-ulkopolitist
otetaan-yhdet
rss-asiastudio
aihe
rikosmyytit
rss-hyvaa-huomenta-bryssel
rss-merja-mahkan-rahat
rss-kaikki-uusiksi
rss-aijat-hopottaa-podcast
rss-raha-talous-ja-politiikka
viisupodi