Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(541)

UL NO. 443: North Korean Co-workers, UBI Failure?, AI-Groupthink, GPS Spoofing…

Switzerland goes open source, Google keeps cookies, DJI not cancelled, Alzheimer's spray, and more… ➡ Check out Vanta and get $1000 off:vanta.com/unsupervised Subscribe to the newsletter at: https://d...

5 Aug 202433min

A Conversation with Christine Gadsby from BlackBerry

In this conversation, I speak with Christine Gadsby, Head of Product Security Operations Team at BlackBerry. We talk about: The Role of AI in Cybersecurity: AI's real advancements, practical applicat...

5 Aug 202441min

UL NO. 442: Crowdstrike Analysis, Cannabis=Soma?, NK Github SE, AI Weaponry

Chinese Solar Builds, DOJ Domain Seizures, Scattered Spider Arrest, Kaiser AI, and more… ➡ Check out Vanta and get $1000 off:vanta.com/unsupervised Subscribe to the newsletter at: https://danielmiessl...

24 Juli 202432min

UL NO. 441: Substrate, OpenAIs AGI Levels, US Literacy Rates

HackerCamp Approaches, Introducing Substrate, Kaspersky--, Exim/Gitlab Vulns, Personal/Business Branding, and more… ➡ Check out the Autonomous IT Podcast:https://community.automox.com/autonomous-it-po...

21 Juli 202424min

UL NO. 440: RAID (Real World AI Definitions)

Twillio API Dump, North Korea Russia, Funny AI Memes, and more… ➡ Check out Vanta and get $1000 off:vanta.com/unsupervised Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the...

11 Juli 202435min

UL NO. 439: Humans vs. AI in Prediction Markets

Project Metaculus, SSH and Juniper 0-Day, China v. Taiwan, R1 Leaks, and more… ➡ Check out Vanta and get $1000 off:vanta.com/unsupervised Subscribe to the newsletter at: https://danielmiessler.com/sub...

4 Juli 202418min

UL NO. 438: Confusion is a Muse

Sonnet 3.5 Support in Fabric, CISA AI Tabletop exercise, Kaspersky ban, China Invasion Scenario, Langchain disilussionment, more… ➡ Check out Vanta and get $1000 off:vanta.com/unsupervised Subscribe t...

27 Juni 202436min

UL NO. 437: My List of Hard-won Life Lessons

New AUGMENTED Course Date, 3 New Essays, Disgruntled deletions, Scale and Merit, Russia moves to Yuan, and more… ➡ Check out the Autonomous IT Podcast:https://community.automox.com/autonomous-it-podca...

21 Juni 202427min