Using the Smartest AI to Rate Other AI
Unsupervised Learning19 Huhti 2025

Using the Smartest AI to Rate Other AI

In this episode, I walk through a Fabric Pattern that assesses how well a given model does on a task relative to humans. This system uses your smartest AI model to evaluate the performance of other AIs—by scoring them across a range of tasks and comparing them to human intelligence levels.

I talk about:

1. Using One AI to Evaluate Another
The core idea is simple: use your most capable model (like Claude 3 Opus or GPT-4) to judge the outputs of another model (like GPT-3.5 or Haiku) against a task and input. This gives you a way to benchmark quality without manual review.

2. A Human-Centric Grading System
Models are scored on a human scale—from “uneducated” and “high school” up to “PhD” and “world-class human.” Stronger models consistently rate higher, while weaker ones rank lower—just as expected.

3. Custom Prompts That Push for Deeper Evaluation
The rating prompt includes instructions to emulate a 16,000+ dimensional scoring system, using expert-level heuristics and attention to nuance. The system also asks the evaluator to describe what would have been required to score higher, making this a meta-feedback loop for improving future performance.

Note: This episode was recorded a few months ago, so the AI models mentioned may not be the latest—but the framework and methodology still work perfectly with current models.

Subscribe to the newsletter at:
https://danielmiessler.com/subscribe

Join the UL community at:
https://danielmiessler.com/upgrade

Follow on X:
https://x.com/danielmiessler

Follow on LinkedIn:
https://www.linkedin.com/in/danielmiessler

See you in the next one!

Become a Member: https://danielmiessler.com/upgrade

See omnystudio.com/listener for privacy information.

Tämä jakso on lisätty Podme-palveluun avoimen RSS-syötteen kautta eikä se ole Podmen omaa tuotantoa. Siksi jakso saattaa sisältää mainontaa.

Jaksot(541)

News & Analysis | No. 278

News & Analysis | No. 278

New Logo, CISA Pulse Secure VPN Orders, Ransomware Gangs + Stock Shorter Scams, Japanese NASA Hacked by China, Dan Kaminsky, LinkedIn Brittons and China/Russia, Codecov Fallout, Technology News, Conte...

26 Huhti 202112min

News & Analysis | No. 277

News & Analysis | No. 277

CISA FBI and NSA Release Five APT29 Targeted Vulnerabilities, FBI Benign Hacking, The US Sanctioned Russia and Expelled Diplomats, Google's Cookie Replacement Not Going Well, NERC Says 1/4 Customers D...

19 Huhti 202127min

News & Analysis | No. 276

News & Analysis | No. 276

Social Media Scraping Outbreak, Microsoft AI Security Tool, FBI/CISA FortiOS Warning, Zoom Vuln at Pwn2Own, AWS Bombing, 485% Ransomware Increase, Technology News, Human News, Ideas Trends & Analysis,...

12 Huhti 202126min

News & Analysis | No. 275

News & Analysis | No. 275

University Accellion Breaches, 533 million Facebook Users' Data, Solarwinds Hackers Got Top DHS Emails, Github Secrets Scanning, Ubiquiti's Breach, Seoul's IoT Towers, Technology News, Human News, Ide...

5 Huhti 202125min

Interview: Amir Majidimehr, Audiophile Industry Disruptor

Interview: Amir Majidimehr, Audiophile Industry Disruptor

In this standalone episode I’m speaking with Amir Majidimehr. Amir is an audiophile, but he has a unique approach to the hobby that’s literally disrupting the industry. He’s basically introduced measu...

2 Huhti 20211h 12min

News & Analysis | No. 274

News & Analysis | No. 274

Securing the Grid, PHP hacked, Russia/China Wargames, China v. Tesla, Top 10 American Threats, Technology News, Human News, Ideas Trends & Analysis, Discovery, Recommendations, and the Weekly Aphorism...

29 Maalis 202120min

The Consumer Authentication Strength Maturity Model (CASMM)

The Consumer Authentication Strength Maturity Model (CASMM)

A maturity model for seeing where a user's internet hygiene currently is, and how to improve it.Become a Member: https://danielmiessler.com/upgradeSee omnystudio.com/listener for privacy information.

25 Maalis 202113min

News & Analysis | No. 273

News & Analysis | No. 273

US Intelligence Says Putin and Russia Tampered in 2020 Election, Finland Says APT31 Hacked Parliament, Google Releases Chrome Data Gathering Report, Ulysses Tracks Cars Worldwide, Twitter Steganograph...

22 Maalis 202121min