Generative Benchmarking with Kelly Hong - #728

Generative Benchmarking with Kelly Hong - #728

In this episode, Kelly Hong, a researcher at Chroma, joins us to discuss "Generative Benchmarking," a novel approach to evaluating retrieval systems, like RAG applications, using synthetic data. Kelly explains how traditional benchmarks like MTEB fail to represent real-world query patterns and how embedding models that perform well on public benchmarks often underperform in production. The conversation explores the two-step process of Generative Benchmarking: filtering documents to focus on relevant content and generating queries that mimic actual user behavior. Kelly shares insights from applying this approach to Weights & Biases' technical support bot, revealing how domain-specific evaluation provides more accurate assessments of embedding model performance. We also discuss the importance of aligning LLM judges with human preferences, the impact of chunking strategies on retrieval effectiveness, and how production queries differ from benchmark queries in ambiguity and style. Throughout the episode, Kelly emphasizes the need for systematic evaluation approaches that go beyond "vibe checks" to help developers build more effective RAG applications. The complete show notes for this episode can be found at https://twimlai.com/go/728.

Jaksot(779)

A Linear-Time Kernel Goodness-of-Fit Test - NIPS Best Paper '17 - TWiML Talk #100

A Linear-Time Kernel Goodness-of-Fit Test - NIPS Best Paper '17 - TWiML Talk #100

In this episode, I speak with Arthur Gretton, Wittawat Jitkrittum, Zoltan Szabo and Kenji Fukumizu, who, alongside Wenkai Xu authored the 2017 NIPS Best Paper Award winner “A Linear-Time Kernel Goodne...

24 Tammi 201822min

Solving Imperfect-Information Games with Tuomas Sandholm - NIPS ’17 Best Paper - TWiML Talk #99

Solving Imperfect-Information Games with Tuomas Sandholm - NIPS ’17 Best Paper - TWiML Talk #99

In this episode I speak with Tuomas Sandholm, Carnegie Mellon University Professor and Founder and CEO of startups Optimized Markets and Strategic Machine. Tuomas, along with his PhD student Noam Brow...

22 Tammi 201827min

Separating Vocals in Recorded Music at Spotify with Eric Humphrey - TWiML Talk #98

Separating Vocals in Recorded Music at Spotify with Eric Humphrey - TWiML Talk #98

In today’s show, I sit down with Eric Humphrey, Research Scientist in the music understanding group at Spotify. Eric was at the Deep Learning Summit to give a talk on Advances in Deep Architectures an...

19 Tammi 201827min

Accelerating Deep Learning with Mixed Precision Arithmetic with Greg Diamos - TWiML Talk #97

Accelerating Deep Learning with Mixed Precision Arithmetic with Greg Diamos - TWiML Talk #97

In this show I speak with Greg Diamos, senior computer systems researcher at Baidu. Greg joined me before his talk at the Deep Learning Summit, where he spoke on “The Next Generation of AI Chips.” Gre...

17 Tammi 201839min

Composing Graphical Models With Neural Networks with David Duvenaud - TWiML Talk #96

Composing Graphical Models With Neural Networks with David Duvenaud - TWiML Talk #96

In this episode, we hear from David Duvenaud, assistant professor in the Computer Science and Statistics departments at the University of Toronto. David joined me after his talk at the Deep Learning S...

15 Tammi 201835min

Embedded Deep Learning at Deep Vision with Siddha Ganju - TWiML Talk #95

Embedded Deep Learning at Deep Vision with Siddha Ganju - TWiML Talk #95

In this episode we hear from Siddha Ganju, data scientist at computer vision startup Deep Vision. Siddha joined me at the AI Conference a while back to chat about the challenges of developing deep lea...

12 Tammi 201834min

Neuroevolution: Evolving Novel Neural Network Architectures with Kenneth Stanley - TWiML Talk #94

Neuroevolution: Evolving Novel Neural Network Architectures with Kenneth Stanley - TWiML Talk #94

Today, I'm joined by Kenneth Stanley, Professor in the Department of Computer Science at the University of Central Florida and senior research scientist at Uber AI Labs. Kenneth studied under TWiML Ta...

11 Tammi 201845min

A Quantum Computing Primer and Implications for AI with Davide Venturelli - TWiML Talk #93

A Quantum Computing Primer and Implications for AI with Davide Venturelli - TWiML Talk #93

Today, I'm joined by Davide Venturelli, science operations manager and quantum computing team lead for the Universities Space Research Association’s Institute for Advanced Computer Science at NASA Ame...

8 Tammi 201834min

Suosittua kategoriassa Politiikka ja uutiset

aikalisa
rss-ootsa-kuullut-tasta
tervo-halme
ootsa-kuullut-tasta-2
politiikan-puskaradio
viisupodi
et-sa-noin-voi-sanoo-esittaa
otetaan-yhdet
rss-podme-livebox
rss-vaalirankkurit-podcast
rss-asiastudio
the-ulkopolitist
rss-kaikki-uusiksi
rss-tekkipodi
io-techin-tekniikkapodcast
rikosmyytit
rss-mina-ukkola
rss-fingo-podcast
rss-hyvaa-huomenta-bryssel
rss-merja-mahkan-rahat