Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(541)

Most Companies Aren't Anywhere Near Ready for AI

Most Companies Aren't Anywhere Near Ready for AI. It's not that companies aren't using AI—it's that they can't.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privac...

3 Mai 5min

We're All Building a Single Digital Assistant

There's tons of confusion about what we're all building towards with Personal AI. Are we building Agents? AI Harnesses? To what end? In this video I lay why I think we're all heading towards a single ...

15 Apr 32min

Why AI Will Replace Knowledge Workers

A longer form discussion on exactly how and why AI will replace knowledge workers.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

21 Mar 1h 16min

Why I Believe in SOTA Models Over Custom Ones

I think the future is cheaper and Open Source SOTA models combined with context, not custom, narrow models.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy in...

11 Mar 1min

AI Quality Inversion

A troubling thought about what we will think about high-quality content in the future. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

6 Mar 1min

The Great Transition

There are a bunch of different transitions happening right now—all at the same time, all (I think) heading in the same direction. Here is a long-form exploration of the various pieces.Become a Member:...

28 Feb 1h 24min

Starting 2026

A welcome back and early entry into 2026. Sponsored by: Knocknoc!Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

30 Jan 25min

Judge AI based on Output, Not Mechanism

How we can use an output-based system to judge whether or not different kinds of technology achieve understanding or intelligence. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com...

22 Nov 20256min