Using the Smartest AI to Rate Other AI
Unsupervised Learning19 Huhti 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(541)

Unsupervised Learning: Episode 35

Unsupervised Learning: Episode 35

[ Subscribe to the Podcast: iTunes | Android | RSS ] News [ ] The hack of Mossak Fonseca has been tied to a breach of their wordpress install through a plugin called Revolution Slider, leading to the ...

11 Huhti 201627min

5 Increasingly Effective Ways to Achieve Immortality

5 Increasingly Effective Ways to Achieve Immortality

[ Subscribe to the Podcast: iTunes | Android | RSS ] — I think a lot about how to become immortal. More than I should, probably. Many think it’s a waste of time. Everyone dies, and it’s foolish to ...

7 Huhti 201613min

Unsupervised Learning: Episode 33

Unsupervised Learning: Episode 33

News [ ] Panama Papers leak [ ] Hackers targeting major US law firms [ ] Ubuntu has some kernel vuln patches out [ ] 50 million turkish citizens have their information dumped online [ ] Microsoft mak...

7 Huhti 201637min

T1SP: Episode 32

T1SP: Episode 32

[ Subscribe to the Podcast: iTunes | Android | RSS ] News * [ ] Verizon Enterprise Solutions had a major data breach of their customer data. This is the group that handles breaches for their custom...

28 Maalis 201636min

T1SP: Episode 31

T1SP: Episode 31

[ Subscribe to the Podcast: iTunes | Android | RSS ] News [ ] FBI saying it will force Apple to hand over source code and signing ability if they don’t comply | http://thehackernews.com/2016/03/fbi-ap...

14 Maalis 201632min

My Response to Sam Harris on the Apple Encryption Debate

My Response to Sam Harris on the Apple Encryption Debate

[ Subscribe to the Podcast: iTunes | Android | RSS ] [ UPDATE: Much credit to Sam for engaging in the conversation. I’m not sure how people claim he’s closed on this topic when he is clearly open to ...

28 Helmi 201636min

T1SP: Episode 29

T1SP: Episode 29

[ Subscribe to the Podcast: iTunes | Android | RSS ] News * [ ] Apple calls out FBI on iPhone decryption case * [ ] Trump calls for a boycott of Apple, from an iPhone * [ ] Judge Rules FBI Must Rev...

23 Helmi 201619min

T1SP: Episode 28

T1SP: Episode 28

[ Subscribe to the Podcast: iTunes | Android | RSS ] News [ ] Major Cisco ASA buffer overflow; patch now [ ] Critical patches for Windows and Flash [ ] The FBI is officially investigating Hillary Clin...

15 Helmi 201642min