Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(541)

The 4 AAAAs of the AI ECOSYSTEM: Assistants, APIs, Agents, and Augmented Reality

In this episode, I break down what I believe is the emerging structure of the AI-powered world we're all building—consciously or not. I call it the “Four A’s”: Assistants, APIs, Agents, and Augmented ...

22 Apr 202527min

A Conversation with Patrick Duffy from Material Security

➡ Secure what your business is made of with Martial Security: https://material.security/ In this episode, I speak with Patrick Duffy from Material Security about modern approaches to email and cloud w...

15 Apr 202526min

AICAD: Artificial Intelligence Capabilities For Attack & Defense

AI is changing cybersecurity at a fundamental level—but how do we decide what to build, and when? In this episode, I outline a structured way to think about AI for security: from foundational ideas to...

12 Apr 202542min

A Possible Path to ASI

The conversation around AGI and ASI is louder than ever—but the definitions are often abstract, technical, and disconnected from what actually matters. In this episode, I break down a human-centered w...

8 Apr 202510min

A Conversation With Matt Muller From Tines

➡ Build, run, and monitor workflows with Tines at: tines.com In this episode, I speak with Matt Muller, Field CSCO at Tines, about how automation and AI are transforming security operations at scale. ...

1 Apr 202539min

UL NO. 474 | Signal OPSEC, White-box Red-teaming LLMs, Unified Company Context (UCC), New Book Recommendations, Single Apple Note Technique, and much more...

STANDARD EDITION: Signal OPSEC, White-box Red-teaming LLMs, Unified Company Context (UCC), New Book Recommendations, Single Apple Note Technique, and much more... You are currently listening to the St...

31 Mar 202518min

A Conversation With Slava Konstantinov From ThreatLocker

➡ Allow what you need, block everything else with ThreatLocker: threatlocker.com In this episode, I speak with Slava Konstantinov, ThreatLocker's MacOS Lead Architect, about their zero-trust approach ...

18 Mar 202533min