Generative Benchmarking with Kelly Hong - #728

Generative Benchmarking with Kelly Hong - #728

In this episode, Kelly Hong, a researcher at Chroma, joins us to discuss "Generative Benchmarking," a novel approach to evaluating retrieval systems, like RAG applications, using synthetic data. Kelly explains how traditional benchmarks like MTEB fail to represent real-world query patterns and how embedding models that perform well on public benchmarks often underperform in production. The conversation explores the two-step process of Generative Benchmarking: filtering documents to focus on relevant content and generating queries that mimic actual user behavior. Kelly shares insights from applying this approach to Weights & Biases' technical support bot, revealing how domain-specific evaluation provides more accurate assessments of embedding model performance. We also discuss the importance of aligning LLM judges with human preferences, the impact of chunking strategies on retrieval effectiveness, and how production queries differ from benchmark queries in ambiguity and style. Throughout the episode, Kelly emphasizes the need for systematic evaluation approaches that go beyond "vibe checks" to help developers build more effective RAG applications. The complete show notes for this episode can be found at https://twimlai.com/go/728.

Jaksot(779)

Ray: A Distributed Computing Platform for Reinforcement Learning with Ion Stoica - TWiML Talk #55

Ray: A Distributed Computing Platform for Reinforcement Learning with Ion Stoica - TWiML Talk #55

The show you’re about to hear is part of a series of shows recorded in San Francisco at the Artificial Intelligence Conference. In this episode, I talk with Ion Stoica, professor of computer science &...

5 Loka 201728min

Topological Data Analysis with Gunnar Carlsson - TWiML Talk #53

Topological Data Analysis with Gunnar Carlsson - TWiML Talk #53

The show you’re about to hear is part of a series of shows recorded in San Francisco at the Artificial Intelligence Conference. My guest for this show is Gunnar Carlsson, professor emeritus of mathema...

3 Loka 201733min

Bayesian Optimization for Hyperparameter Tuning with Scott Clark - TWiML Talk #50

Bayesian Optimization for Hyperparameter Tuning with Scott Clark - TWiML Talk #50

As you all know, a few weeks ago, I spent some time in SF at the Artificial Intelligence Conference. While I was there, I had just enough time to sneak away and catch up with Scott Clark, Co-Founder a...

2 Loka 201747min

Symbolic and Sub-Symbolic Natural Language Processing with Jonathan Mugan - TWiML Talk #49

Symbolic and Sub-Symbolic Natural Language Processing with Jonathan Mugan - TWiML Talk #49

Like last week’s interview with Bruno Goncalves, this week’s interview was also recorded at the last O’Reilly AI Conference back in New York in June. Also like last week’s show, this week’s is also fo...

25 Syys 201743min

Word2Vec & Friends with Bruno Gonçalves - TWiML Talk #48

Word2Vec & Friends with Bruno Gonçalves - TWiML Talk #48

This week i'm bringing you an interview from Bruno Goncalves, a Moore-Sloan Data Science Fellow at NYU. As you’ll hear in the interview, Bruno is a longtime listener of the podcast. We were able to co...

19 Syys 201732min

Evolutionary Algorithms in Machine Learning with Risto Miikkulainen - TWiML Talk #47

Evolutionary Algorithms in Machine Learning with Risto Miikkulainen - TWiML Talk #47

My guest this week is Risto Miikkulainen, professor of computer science at UT-Austin and vice president of Research at Sentient Technologies. Risto came locked and loaded to discuss a topic that we've...

11 Syys 201758min

Agile Machine Learning with Jennifer Prendki - TWiML Talk #46

Agile Machine Learning with Jennifer Prendki - TWiML Talk #46

My guest this week is Jennifer Prendki. That name might sound familiar, as she was one of the great speakers from my Future of Data Summit back in May. At the time, Jennifer was senior data science ma...

5 Syys 201748min

LSTMs, Plus a Deep Learning History Lesson with Jürgen Schmidhuber - TWiML Talk #44

LSTMs, Plus a Deep Learning History Lesson with Jürgen Schmidhuber - TWiML Talk #44

This week we have a very special interview to share with you! Those of you who’ve been receiving my newsletter for a while might remember that while in Switzerland last month, I had the pleasure of in...

28 Elo 20171h 3min

Suosittua kategoriassa Politiikka ja uutiset

aikalisa
rss-ootsa-kuullut-tasta
tervo-halme
ootsa-kuullut-tasta-2
politiikan-puskaradio
viisupodi
rss-podme-livebox
otetaan-yhdet
et-sa-noin-voi-sanoo-esittaa
rss-vaalirankkurit-podcast
rss-asiastudio
rss-tasta-on-kyse-ivan-puopolo-verkkouutiset
rss-tekkipodi
io-techin-tekniikkapodcast
linda-maria
the-ulkopolitist
rss-polikulaari-humanisti-vastaa-ja-muut-ts-podcastit
rss-kaikki-uusiksi
rss-hyvaa-huomenta-bryssel
rss-merja-mahkan-rahat