Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(541)

Unsupervised Learning: No. 187

Lots of people in the security community went silly over the FaceApp application last week, basically saying that you shouldn't be using the application because they'll steal your face and then be abl...

22 Jul 201935min

Time Speeds Up When You’re Wasting It

An essay on why time can feel like it's speeding up when you get older, and how to slow it back down.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy informat...

20 Jul 20195min

Unsupervised Learning: No. 186

Parts of Manhattan had a power outage Saturday night, which happened to be the anniversary of another power outage in 1977. The power company apologized but didn't explain what happened. The hacker in...

15 Jul 201920min

Unsupervised Learning: No. 185

The Telegraph has found strong links between Huawei employees and Chinese intelligence agencies. The Huawei counter was that this was extremely common among telecom companies, and that it wasn't a big...

8 Jul 201921min

The World is Collapsing Into Two Countries—Green and Red

The world being sorted into two different countries—a Green country of the top 10% of income/wealk, and a Red country that's everyone else. These countries are separated not by geography, but by class...

3 Jul 20198min

Unsupervised Learning: No. 184

I created a new tutorial on OWASP Amass, and just joined the team as a contributor as well. TutorialChinese hacking groups have been embedded deep inside multiple major US tech firms for many years, i...

1 Jul 201918min

Unsupervised Learning: No. 183

There's a Linux vulnerability called SACK Panic (among other names) that takes advantage of a kernel feature called Selective ACK. The feature lets systems tell the other side of the conversation how ...

24 Jun 201913min

Unsupervised Learning: No. 182

The US is supposedly ramping up attacks against Russian power grid through the use of new cyberattack powers granted by Trump. I am happy to hear of this, but it's an example of where we as outsiders ...

18 Jun 201910min