Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(541)

The Biggest Advantage in Machine Learning Will Come From Superior Coverage, Not Superior Analysis

Many people, in many fields, think Machine Learning won't replace their analysts because their humans are better than an algorithm. But it's not just about side-by-side comparisons. The bigger questio...

3 Jan 20188min

It's Wrong to Fear-monger on IoT Security

How it's shortsighted and irresponsible for InfoSec professionals to fear-monger on IoT Security, and what we should be saying instead.Become a Member: https://danielmiessler.com/upgradeSee omnystudio...

3 Jan 20185min

Unsupervised Learning: No. 106

Swatting death, Ethereum kidnap, Chinese dystopia, Alteryx S3 bucket, Starbucks Monero, Forever21, Microphone ads, technology news, human news, discovery, notes, recommendations, and the aphorism of t...

3 Jan 201828min

Unsupervised Learning: No. 105

TRITON, 1.4 billion credentials, HP keyloggers, iTunes Bitcoin laundering, removing credit card signatures, technologgy news, human news, discovery, notes, recommendations, and the aphorism of the wee...

18 Dec 201723min

Unsupervised Learning: No. 104

NiceHash hacked, Apple bugs, Stealing Cars via Relay, Crypto Collusion, technologgy news, human news, discovery, notes, recommendations, and the aphorism of the week…Become a Member: https://danielmie...

12 Dec 201725min

Unsupervised Learning: No. 103

Uber's mess, Google tracking users, AI finding missiles, drone disclosure, net neutrality, tech news, human news, ideas, discovery, recommendations, aphorism, and more…Become a Member: https://danielm...

27 Nov 201728min

Unsupervised Learning: No. 102

Github security, China IW, Brexit IW, S3 again, Quad9 DNS security, tech news, human news, ideas, discovery, recommendations, aphorism, and more…Become a Member: https://danielmiessler.com/upgradeSee ...

20 Nov 201726min

Unsupervised Learning: No. 101

Verizon’s DBIR Report, sleeping fingerprints, IoT legislation, S3 security tools, AI tricks scammers, SEALs kill Green Beret, tech news, human news, ideas, discovery, recommendations, aphorism, and mo...

13 Nov 201735min