Using the Smartest AI to Rate Other AI
Unsupervised Learning19 Huhti 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(541)

Unsupervised Learning: No. 209

Unsupervised Learning: No. 209

Ring Sued, Mean Time to Hardening, APT20 2FA, China Base Pictures, China Satellites, Angled Toilets, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Ap...

30 Joulu 201915min

Unsupervised Learning: No. 208

Unsupervised Learning: No. 208

Mobile Tracking, Chinese Drone-Flu Terrorism, Message Spying, Bing Misinformation, 23andMe GlaxoSmithKline, Spam Laws, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations,...

23 Joulu 201915min

Unsupervised Learning: No. 207

Unsupervised Learning: No. 207

Pentagon vendor requirements, Ring camera freakout, Bluetooth Thieves, Palantir Pentagon, Amazon Rekognition, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the ...

17 Joulu 201928min

Unsupervised Learning: No. 206

Unsupervised Learning: No. 206

Vietnamese BMW APT, Defense Contractor Prep, China replacing a culture, HackerOne Cookie Snafu, Chinese Also Worried About Privacy, China Mobile Face, CDC Flu Warning, AWS Sagemaker, Technology News, ...

9 Joulu 201921min

Unsupervised Learning: No. 205

Unsupervised Learning: No. 205

Spam trends, CWE's latest 25, Uber audio recordings, Uber unauthorized drivers, Chinese research theft, Google state-actor notifications, bluetooth burglars, Nixon deepface, Technology News, Human New...

2 Joulu 201934min

Unsupervised Learning: No. 203

Unsupervised Learning: No. 203

Google health care, Google checking, Github open source, China policy hack, Hactivist bounties, healthcare attacks, facial protests, OSINT CTF, surveillance robots, Technology News, Human News, Ideas ...

18 Marras 201918min

Unsupervised Learning: No. 202

Unsupervised Learning: No. 202

Capital fired, DHS biodata, Twitter insiders, Baltimore Cyber Insurance, Airbnb Assessment, Google Play Malware, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and t...

11 Marras 201914min

Unsupervised Learning: No. 201

Unsupervised Learning: No. 201

Unify drama, Fancy cheating, NSO lawsuits, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism…Become a Member: https://danielmiessler.com/upgrade...

4 Marras 201919min