Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(541)

UL NO. 431: Companies are Graphs of Algorithms

The US goes skills-based, AI is mostly prompting, simulation -> reality, 30 useful concepts, and more…… Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at:ht...

9 Mai 202411min

UL NO. 430: The Courage to be Disliked

How I use local AI models, MI5 vetting research students, the first AI deepfake racism attack, and more… Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at:h...

3 Mai 202430min

UL NO. 429: Build Your Career Around Problems

Stanford's State of AI, Peter Thiel vs. Tyler Cowen, China Taiwan Hacking Prep, GenZ Outperforming, and more… Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community...

27 Apr 202421min

UL NO. 428: Reason to Fear; Reason to Build.

AI Propaganda, Speaking Events, analhttps://www.linkedin.com/in/danielmiessleryze_presentation Pattern, Guarding Your Energy Reserves, and more… Subscribe to the newsletter at: https://danielmiessler....

23 Apr 202416min

UL NO. 427: AI's Predictable Future

Israeli identity reveal, deepfaked content summaries, Altman/Ive device, wealthy kids, Cowen v. Haidt, and more… Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL commun...

10 Apr 202421min

UL NO. 425: The Efficient Security Principle

US drone combat, extract ideas from any book, Pinker writing analysis, Flipper reversal, GPT-5 updates, and more… Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL commu...

27 Mar 202421min

UL NO. 424: Raising Security's Floor

Insane Video Deepfakes, Devin Gets Slack Access, New Fabric Patterns, AI Application Interfaces, Let Grow, and more… Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL co...

21 Mar 202426min

A Conversation with Jason Meller of Kolide/1Password

In this sponsored conversation, I speak with Jason Meller. Jason is the founder of Kolide, which has just recently been acquired by 1Password. We discuss: - Kolide's acquisition by 1Password- The syne...

19 Mar 202421min