Unsupervised Learning19 Apr 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(541)

Unsupervised Learning: No. 131

Predicting your credit rating based on the tech you use, Russians attack Germans, WPA3, China bird drones, AT&T and Verison to stop selling our location data, Facebook red team, Twitter Smyte, plus te...

26 Juni 201829min

Unsupervised Learning: No. 129

Reboot your router, China hacked a U.S. Navy contractor and stole around 600GB of top secret data. Newark, NJ is monitoring much of the city with surveillance cameras, and they're making the camera fo...

12 Juni 201811min

Unsupervised Learning: No. 128

Pentagon background checks, China using machine learning in schools, Rusian ethnicity detecting AI, US Military presence in Africa, Atlanta lost dashcam footage, Kidnapping insurance, Technology News,...

4 Juni 201810min

Unsupervised Learning: No. 127

VPNFilter botnet, Echo private convo, Ghostery GDPR fail, PornHub VPN, Technology News, Human News, Ideas, Trends, & Analysis, Discovery, Recommendations, the weekly Aphorism, and more…Become a Member...

29 Maj 20189min

Unsupervised Learning: No. 126

VPNFilter botnet, LA + Palantir, Amazon Surveillance, Momentum report, Clapper says Russia turned the election, Chinese supply chain attacks, Tech News, Human News, Ideas, Discovery, Recommendation, t...

25 Maj 20189min

Unsupervised Learning: No. 125

Regulators aren't staffed to audit you on GDPR, inaudible Siri and Alexa commands, iOS 4 is bringing lots of privacy updates, California DNA storage, technology news, human news, Ideas, recommendation...

18 Maj 201812min

If You’re Not Doing Continuous Asset Management You’re Not Doing Security

How enterprises are completely ignoring the security activity that could help the most.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

16 Maj 20187min

Unsupervised Learning: No. 120

It's 2 billion users now, Liinux beep, Digital Shadows finds fail files, cloud misconfiguration, AlterEgo, AI applications, Alexa sending payments, Tech, Ideas, Recommendation, Aphorism, and more…Beco...

9 Apr 201819min