Generative Benchmarking with Kelly Hong - #728

Generative Benchmarking with Kelly Hong - #728

In this episode, Kelly Hong, a researcher at Chroma, joins us to discuss "Generative Benchmarking," a novel approach to evaluating retrieval systems, like RAG applications, using synthetic data. Kelly explains how traditional benchmarks like MTEB fail to represent real-world query patterns and how embedding models that perform well on public benchmarks often underperform in production. The conversation explores the two-step process of Generative Benchmarking: filtering documents to focus on relevant content and generating queries that mimic actual user behavior. Kelly shares insights from applying this approach to Weights & Biases' technical support bot, revealing how domain-specific evaluation provides more accurate assessments of embedding model performance. We also discuss the importance of aligning LLM judges with human preferences, the impact of chunking strategies on retrieval effectiveness, and how production queries differ from benchmark queries in ambiguity and style. Throughout the episode, Kelly emphasizes the need for systematic evaluation approaches that go beyond "vibe checks" to help developers build more effective RAG applications. The complete show notes for this episode can be found at https://twimlai.com/go/728.

Jaksot(779)

Differential Privacy Theory & Practice with Aaron Roth - TWiML Talk #132

Differential Privacy Theory & Practice with Aaron Roth - TWiML Talk #132

In the first episode of our Differential Privacy series, I'm joined by Aaron Roth, associate professor of computer science and information science at the University of Pennsylvania. Aaron is first and...

30 Huhti 201842min

Optimal Transport and Machine Learning with Marco Cuturi - TWiML Talk #131

Optimal Transport and Machine Learning with Marco Cuturi - TWiML Talk #131

In this episode, i’m joined by Marco Cuturi, professor of statistics at Université Paris-Saclay. Marco and I spent some time discussing his work on Optimal Transport Theory at NIPS last year. In our d...

26 Huhti 201832min

Collecting and Annotating Data for AI with Kiran Vajapey - TWiML Talk #130

Collecting and Annotating Data for AI with Kiran Vajapey - TWiML Talk #130

In this episode, I’m joined by Kiran Vajapey, a human-computer interaction developer at Figure Eight. In this interview, Kiran shares some of what he’s has learned through his work developing applicat...

23 Huhti 201840min

Autonomous Aerial Guidance, Navigation and Control Systems with Christopher Lum - TWiML Talk #129

Autonomous Aerial Guidance, Navigation and Control Systems with Christopher Lum - TWiML Talk #129

Ok, In this episode, I'm joined by Christopher Lum, Research Assistant Professor in the University of Washington’s Department of Aeronautics and Astronautics. Chris also co-heads the University’s Auto...

19 Huhti 201852min

Infrastructure for Autonomous Vehicles with Missy Cummings - TWiML Talk #128

Infrastructure for Autonomous Vehicles with Missy Cummings - TWiML Talk #128

In this episode, I’m joined by Missy Cummings, head of Duke University’s Humans and Autonomy Lab and professor in the department of mechanical engineering. In addition to being an accomplished researc...

16 Huhti 201843min

Hyper-Personalizing the Customer Experience w/ AI with Rob Walker - TWiML Talk #127

Hyper-Personalizing the Customer Experience w/ AI with Rob Walker - TWiML Talk #127

In this episode, we're joined by Rob Walker, Vice President of decision management and analytics at Pegasystems, a leading provider of software for customer engagement and operational excellence. Rob ...

12 Huhti 201841min

Information Extraction from Natural Document Formats with David Rosenberg - TWiML Talk #126

Information Extraction from Natural Document Formats with David Rosenberg - TWiML Talk #126

In this episode, I’m joined by David Rosenberg, data scientist in the office of the CTO at financial publisher Bloomberg, to discuss his work on “Extracting Data from Tables and Charts in Natural Docu...

9 Huhti 201845min

Human-in-the-Loop AI for Emergency Response & More w/ Robert Munro - TWiML Talk #125

Human-in-the-Loop AI for Emergency Response & More w/ Robert Munro - TWiML Talk #125

In this episode, I chat with Rob Munro, CTO of the newly branded Figure Eight, formerly known as CrowdFlower. Figure Eight’s Human-in-the-Loop AI platform supports data science & machine learning team...

5 Huhti 201848min

Suosittua kategoriassa Politiikka ja uutiset

aikalisa
rss-ootsa-kuullut-tasta
tervo-halme
ootsa-kuullut-tasta-2
politiikan-puskaradio
viisupodi
et-sa-noin-voi-sanoo-esittaa
otetaan-yhdet
rss-vaalirankkurit-podcast
rss-asiastudio
rss-podme-livebox
the-ulkopolitist
rss-kaikki-uusiksi
rss-tekkipodi
io-techin-tekniikkapodcast
rikosmyytit
rss-mina-ukkola
rss-fingo-podcast
rss-hyvaa-huomenta-bryssel
rss-kuka-mina-olen