Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(541)

4 Things To Do in the First Week of Every January

A short piece on why I don't like New Years resolutions, and the four things I prefer to do instead. Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy informat...

27 Dec 20162min

Unsupervised Learning: No. 58

This week's topics: Yahoo!, Shadowbrokers, Building Your Own Honeytrapping Infrastructure, The Power of Newsletters, Project Aristotle, and more…Become a Member: https://danielmiessler.com/upgradeSee ...

19 Dec 201614min

Unsupervised Learning: No. 57

This week’s topics: Russia gave us attribution for Christmas, the NSA is shedding talent, the evilest ransomware, how to raise someone's IQ in 2 minutes, and more…Become a Member: https://danielmiessl...

12 Dec 201629min

Unsupervised Learning: No. 56

Gooligan, Korean Game Hacking Law, DoubleFlag Experian Hack, Georgia Tech Attribution Research, Amazon's re:Invent Conference Highlights, recommended links, and more…Become a Member: https://danielmie...

5 Dec 201614min

Unsupervised Learning: No. 55

Biowarfare defenses, AI advances, mergers and acquisitions, Facebook and censorship, IoT definitions, the philosophy of Westworld, and more...Become a Member: https://danielmiessler.com/upgradeSee omn...

28 Nov 201624min

The Difference Between Threats, Threat Actors, Vulnerabilities, and Risks

My essay that clearly separates the differences between the core infosec terms of threat, threat actor, vulnerabilty, and risk.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/lis...

27 Nov 20165min

The Difference Between Existentialism, Nihilism, and Absurdism

For centuries there have been people who believe there is no intrinsic meaning in the universe. Here I’ll summarize the three major branches of this belief, and how each proposes we deal with the situ...

22 Nov 20168min

Stop Being Proud of Complexity

An essay on how complexity often communicates the exact opposite of its desired effect.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

22 Nov 20163min