Using the Smartest AI to Rate Other AI
Unsupervised Learning19 Huhti 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(541)

A Conversation with Rob Allen from ThreatLocker

A Conversation with Rob Allen from ThreatLocker

In this conversation, I speak with Rob Allen, Chief Product Officer at ThreatLocker. We talk about: ThreatLocker’s Unique Zero Trust Approach to Cybersecurity:How ThreatLocker’s "deny by default, perm...

18 Marras 202432min

UL NO. 458: Ollama Vulnerabilities, Rating AI Using AI, The Mantis Hack-back Framework

UL NO. 458: Ollama Vulnerabilities, Rating AI Using AI, The Mantis Hack-back Framework

My conversation with Jason Haddix from Flare, Google finds a Zero-Day with AI, Robot Dogs Protecting Mar-a-Lago, and more... Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join t...

17 Marras 202432min

A Conversation with Jason Haddix from Flare

A Conversation with Jason Haddix from Flare

Streamline Your Cybersecurity with Flare Here: https://try.flare.io/unsupervised-learning/ In this conversation, I speak with Jason Haddix, founder of Arcanum Security and CISO at Flare. We talk about...

11 Marras 202430min

UL NO. 454: The First AI Breaches

UL NO. 454: The First AI Breaches

AI Avatar Breaches, Gullibility is Vulnerability: Conspiracy is Threat, Caldera's New Plugin, and more... Try Out the ThreatLocker to take your security to the next level: https://www.threatlocker.com...

18 Loka 202435min

How My Projects Fit Together (Substrate, Fabric, Telos, Daemon, and Human 3.0)

How My Projects Fit Together (Substrate, Fabric, Telos, Daemon, and Human 3.0)

This episode, "How My Projects Fit Together," is a follow-up to a previous post called "What I Am Doing & How It's Going". Here, Daniel Miessler addresses the most commonly asked questions: "I see all...

15 Loka 20241h 1min

Human 3.0—The Skills & Mental Frames Required To Thrive In An AI World

Human 3.0—The Skills & Mental Frames Required To Thrive In An AI World

Human 3.0 is here. In this conference for the United Nations, Daniel Miessler introduces the topic of Human 3.0 philosophy and the skills and mental frameworks needed to thrive in an AI-driven world. ...

9 Loka 202430min

UL NO. 452: The New Hotness: NotebookLM

UL NO. 452: The New Hotness: NotebookLM

China prepping for kinetic using cyber?, Automatic podcast creation using NotebookLM, VM + AI, and more... Subscribe to the newsletter at: https://danielmiessler.com/subscribe Join the UL community at...

7 Loka 202450min

NotebookLM Podcast: David Deutsch, Understanding, and AI

NotebookLM Podcast: David Deutsch, Understanding, and AI

This is a NotebookLM podcast based on a long conversation I had with my AI, DARSA, on the topic of whether AIs truly understand things and/or are capable of creativity.Become a Member: https://danielm...

2 Loka 202412min